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Executive  Summary 


This  report  details  the  approach  and  key  elements  of  conducting  pilot  tests  to  pro¬ 
vide  enhanced  incentives  for  Department  of  Defense  (DoD)  acquisition  profes¬ 
sionals  on  Acquisition  Category  I  (ACAT I)  program  management  teams.  The 
objective  of  the  tests  is  to  determine  whether  these  incentives  can  contribute  to 
program  success  in  terms  of  cost,  schedule,  and  performance. 

Background 

Section  5001  (b)  of  the  Federal  Acquisition  Streamlining  Act  (FASA)  requires  the 
Secretary  of  Defense  to  review  the  incentives  and  personnel  actions  available  to 
DoD  to  encourage  excellence  in  the  management  of  defense  acquisition  programs, 
and  to  provide  a  system  of  enhanced  incentives  to  facilitate  the  achievement  of 
cost,  schedule,  and  performance  goals.  The  system  is  intended  to  relate  pay  to  per¬ 
formance  and  to  provide  for  consideration  (in  personnel  evaluations  and  promo¬ 
tion  decisions)  of  the  extent  to  which  personnel  performance  contributes  to 
achieving  goals  established  for  major  defense  acquisition  programs. 

Test  Design 

The  test  of  any  reward  system  is  whether  it  encourages  behavior  that  improves 
performance  and  then  rewards  the  results  of  that  behavior.  The  test  we  have  de¬ 
signed  is  intended  to  measure  the  impact  of  workforce  incentives  on  program  suc¬ 
cess  while  neutralizing  the  effects  of  other  factors  known  to  influence  the 
acquisition  program.  We  will  compare  test  programs  that  use  enhanced  incentives 
with  control  programs  that  do  not  to  determine  whether  the  incentives  can  explain 
differences  in  program  performance. 

Stakeholder  Findings 

We  conducted  an  extensive  review  of  government  and  industry  initiatives  relating 
to  performance  management  strategies  and  their  effect  on  employee  motivation 
and  performance.  We  met  with  subject-matter  experts  from  DoD,  the  Office  of 
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Personnel  Management,  and  the  private  sector,  including  civilian  personnel  spe¬ 
cialists,  military  compensation  mangers,  senior  acquisition  leaders,  management 
consulting  experts,  and  technology  integration  professionals.  Additionally,  we 
surveyed  nearly  1,500  acquisition  professionals,  including  both  military  and  ci¬ 
vilian  as  well  as  science,  engineering,  technical,  and  administrative  (SETA)  sup¬ 
port  contractors  in  the  program  offices  and  senior  acquisition  students  at  the 
Industrial  College  of  the  Armed  Forces  and  the  Defense  Systems  Management 
College.  The  purpose  of  the  survey  was  to  determine  which  rewards  are  valued 
most,  what  the  effectiveness  of  current  performance  management  practices  is,  and 
which  critical  processes  are  within  the  respondents’  control. 

The  results  of  this  research  indicate  that  rewards  and  recognition  have  a  large  po¬ 
tential  to  influence  acquisition  program  success.  If  rewards  are  to  be  used  success¬ 
fully  to  motivate  people  and  drive  performance  in  this  environment,  acquisition 
professionals  at  all  levels  told  us: 

♦  The  team  succeeds  or  fails  collectively,  and  should  be  rewarded  collec¬ 
tively.  Therefore,  all  team  members  should  be  eligible  to  participate  in  the 
program,  including  civilians,  military,  and  SETA  support  contractors. 

♦  Rewards  should  be  team-based,  with  an  opportunity  for  the  team  to  recog¬ 
nize  individuals. 

♦  Rewards  must  be  equitable,  and  either  cash  awards  or  savings  bonds 
should  be  available  to  all  team  members. 

♦  Current  performance  management  systems  lack  credibility  and  do  not 
drive  performance.  A  credible  performance  measurement  system  must  be 
established  to  which  incentives  can  then  be  linked. 

Implementation 

The  implementation  of  the  test  in  ACAT I  program  offices  should  be  accom¬ 
plished  in  two  stages.  Our  research  has  shown  that  a  credible  team-based  perform¬ 
ance  management  system  must  be  in  place  before  incentives  are  introduced.  The 
survey  provided  evidence  that  existing  performance  appraisal  systems  are  not  a 
credible  measure  of  job  performance  and  do  not  effectively  measure  team  per¬ 
formance. 

Stage  one  would  establish  a  team-based  performance  management  system.  After 
an  extensive  review  of  government  and  industry  initiatives  in  the  area  of  perform¬ 
ance  management  strategies,  we  determined  that  the  Balanced  Scorecard  repre¬ 
sents  the  leading  edge  of  management  thinking  over  the  past  five  years.  The 
Balanced  Scorecard  provides  a  framework  for  communicating  organizational 
strategy,  measuring  performance  against  clearly  defined  goals  aligned  with  strat¬ 
egy,  and  linking  rewards  to  focus  behavior  and  drive  performance.  By  clearly 
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defining  desired  outcomes  and  the  drivers  of  those  outcomes,  senior  managers  can 
channel  the  energies,  abilities,  and  specific  knowledge  of  people  throughout  the 
organization  toward  achieving  long-term  goals. 

Stage  two  would  begin  after  the  performance  management  system  is  implemented 
and  employees  and  management  feel  comfortable  with  and  have  confidence  in  the 
strategic  objectives,  measures,  and  targets  represented  in  their  scorecard.  At  this 
point,  incentives  will  be  linked  to  stretch  targets  related  to  the  strategic  objectives. 
The  purpose  of  incentives  is  to  motivate,  drive  performance,  and  focus  the  entire 
team  on  the  common  mission  and  strategy-based  objectives. 

Measuring  Response 

This  test  design  measures  program  success  using  the  existing  reporting  system, 
Selected  Acquisition  Reports,  and  existing  baselines  from  the  Acquisition  Pro¬ 
gram  Baseline.  A  single-factor  analysis  of  variance  (ANOVA)  model  will  be  used 
to  analyze  the  test  data  to  determine  whether  there  are  statistical  differences  in  the 
program  success  measures  among  the  test  programs.  The  primary  measures  of 
program  success  will  be  cost  and  schedule  variances.1  Although  our  analysis  indi¬ 
cates  that  there  has  been  some  reductions  in  cost  variances,  programs  still  average 
17  percent  overruns  after  inflation  and  quantity  change  adjustments.2 

Recommendations 

We  recommend  that  16  pilot  programs  participate  in  the  test  to  determine  whether 
incentives  can  have  a  positive  impact  on  program  success.  This  is  the  minimum 
required  for  collecting  enough  data  to  detect  statistical  differences  among  the  test 
programs.  In  stage  one,  all  pilot  programs  should  have  successfully  implemented 
the  common  performance  measurement  system — the  Balanced  Scorecard — to 
which  incentives  will  be  linked  in  stage  two.  The  programs  will  be  divided  into 
four  groups,  each  testing  one  of  four  levels  of  incentives.  The  first  group  will  be 
the  control  group  and  use  only  existing  incentives.  The  second  group  would  have 
a  fenced  awards  budget  to  test  an  enhanced  system  of  nonmonetary  awards  linked 
to  team  performance.  The  third  and  fourth  groups  would,  respectively,  provide 
smaller  and  larger  monetary  awards,  the  larger  amount  testing  the  notion  that 
larger  rewards  lead  to  larger  performance  gains. 

The  test  period  should  be  five  years  to  allow  time  for  the  interventions  to  affect 
program  cost,  schedule,  and  performance  outcomes  and  to  observe  the  results. 


1  Research  has  shown  that  acquisition  programs  usually  meet  their  systems’  technical  perform¬ 
ance  objectives  usually  at  the  expense  of  cost  or  schedule  variations. 

2  The  Defense  System  Cost  Performance  Database,  Cost  Growth  Analysis  Using  Selected  Ac¬ 
quisition  Reports,  J.M.  Jarvaise,  J.A.  Drezner,  D.  Norton,  RAND,  National  Defense  Research  In¬ 
stitute,  1996. 
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Congressionally  authorized  personnel  demonstration  projects  authorize  five-year 
tests  to  observe  changes  in  the  workforce  over  time. 

Conclusion 

As  a  result  of  acquisition  reform,  employees  are  being  encouraged  to  take  responsi¬ 
bility  for  their  individual  growth  and  advancement,  to  perform  more  diverse  tasks, 
and  to  be  more  creative  when  challenges  arise.  Innovative  companies  are  developing 
programs  to  focus  employees  on  their  strategic  objectives  and  motivate  breakthrough 
improvements  in  such  critical  areas  as  product,  process,  and  customer  development. 
These  programs  include  reengineered  performance  management  and  incentive  pay 
plans,  emphasizing  the  link  between  pay  and  performance. 

This  test  satisfies  FASA’s  requirement  to  provide  a  system  of  enhanced  incentives  to 
facilitate  the  achievement  of  cost,  schedule,  and  performance  goals.  Using  pilot 
programs  to  test  the  hypothesis  that  incentives  improve  the  likelihood  of  program 
success,  offers  an  opportunity  to  gain  insight  and  to  leverage  lessons  learned.  The  test 
design  provides  two  things:  evidence  for  determining  the  potential  of  incentives  to 
improve  performance,  and  a  framework  for  implementing  pay-for-performance 
efficiently  and  on  a  larger  scale. 
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Chapter  1 

Introduction 


Purpose 

This  report  explains  the  approach  to  and  key  elements  of  conducting  pilot  tests  to 
provide  enhanced  incentives  for  Department  of  Defense  (DoD)  acquisition  profes¬ 
sionals  on  Acquisition  Category  I  (ACAT I)  program  management  teams  (PMTs).1 
The  objective  of  the  test  is  to  determine  whether  these  incentives  can  contribute  to 
program  success  as  measured  by  cost,  schedule,  and  performance  goals. 

The  Requirement 

Section  5001  (b)  of  the  Federal  Acquisition  Streamlining  Act  (FASA)  of  1994  re¬ 
quires  the  Secretary  of  Defense  to  review  the  incentives  and  personnel  actions  avail¬ 
able  to  DoD  to  encourage  excellence  in  the  management  of  defense  acquisition 
programs,  and  to  provide  a  system  of  enhanced  incentives  to  facilitate  the  achieve¬ 
ment  of  cost,  schedule,  and  performance  goals.  The  system  is  intended  to  relate  pay  to 
performance  and  to  provide  for  consideration  (in  personnel  evaluations  and  promotion 
decisions)  of  the  extent  to  which  personnel  performance  contributes  to  achieving  the 
program’s  cost,  schedule,  and  performance  goals. 

DoD’s  Response 

In  response  to  the  FASA  requirement,  the  Deputy  Under  Secretary  of  Defense  for 
Acquisition  Reform  (DUSD[AR])  asked  the  Logistics  Management  Institute  (LMI) 
to  examine  personnel  evaluation  practices  and  incentives  currently  available  to  the 
Department,  and  to  research  private  industry  and  government  best  practices.  LMI’s 
initial  findings  were  documented  in  an  interim  report  that  assessed  the  acquisition 
environment  and  identified  desirable  characteristics  of  effective  incentive  systems.2 


1  ACAT  I  programs  are  defined  by  the  Under  Secretary  of  Defense  for  Acquisition  and  Tech¬ 
nology  (USD[A&T])  as  those  estimated  to  require  eventual  expenditure  of  more  than  $355  million 
(FY96  constant  dollars)  for  research,  development,  test,  and  evaluation  or  more  than  $2,135  billion 
(FY96  constant  dollars)  for  procurement,  or  those  designated  by  the  USD(A&T)  to  be  ACAT  I. 

2  LMI  Report  AQ505LN1,  Incentives  for  Acquisition  Personnel,  Anthony  Durso,  Albert 
Schroetel,  et  al.,  November  1995. 
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Effective  incentive  systems 

♦  are  linked  to  the  organization’s  strategic  objectives, 

♦  involve  employees  in  the  design  to  establish  ownership, 

♦  are  open  and  easily  understood, 

♦  have  clear  linkage  to  performance  (line  of  sight), 

♦  include  meaningful  levels  of  rewards  (at  least  10  percent  of  pay), 

♦  cover  meaningful  periods  of  time, 

♦  are  linked  to  credible  appraisal  systems, 

♦  emphasize  group  performance  but  allow  the  team  to  recognize  individuals, 

♦  use  quantitative  measures  whenever  possible,  and 

♦  set  achievable,  yet  challenging,  goals. 

The  DUSD(AR)  prepared  a  response  to  Congress  which  summarized  the  interim 
report  and  requested  authority  to  conduct  pilot  tests  of  enhanced  incentives  for 
PMTs,  including  DoD  civilians,  military  personnel,  and  science,  engineering, 
technical,  and  administrative  (SETA)  contract  employees  in  direct  support  of  pro¬ 
gram  offices.  The  response  recommended  that  program  savings  be  made  available 
to  fund  performance  bonuses  (to  facilitate  and  cement  the  relationship  between 
program  savings  and  improved  performance)  and  that  payment  of  monetary  in¬ 
centives  include  military  members  of  the  acquisition  program  team.3 

Other  Catalysts  for  Incentives 

Legislation  and  Policy 

Congress  has  repeatedly  expressed  its  desire  to  shift  from  evaluating  inputs  to 
measuring  outcomes  in  the  conduct  of  acquisition  programs.  In  addition  to  FASA 
and  its  workforce  performance  enhancement  provisions,  the  Federal  Acquisition 
Reform  (Clinger-Cohen)  Act  of  1996  directs  the  head  of  each  executive  agency  to 
establish  policies  and  procedures  for  the  effective  management  (including  acces¬ 
sion,  education,  training,  career  development,  and  performance  incentives)  of  the 
acquisition  workforce.  Model  legislation  has  been  drafted  for  creating  perform¬ 
ance-based  organizations  (PBOs)  which  recognizes  the  potential  of  the  reward 


3  There  is  no  legislative  proposal  to  date  that  provides  the  authority  for  monetary  performance 
awards  for  the  military. 
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system  to  affect  performance.4  The  language  gives  PBOs  broad  discretion  to  de¬ 
sign  leading  edge  reward  systems  that  are  as  aggressive  as  any  in  the  private  sec¬ 
tor.5  It  builds  on  the  1993  Government  Performance  and  Results  Act  (GPRA) 
goal-setting  and  performance  measurement  requirements,  authorizing  the  creation 
of  awards  programs  based  on  organizational,  group,  and  individual  achievements. 

DoD  5000  series  directives  specifically  address  the  use  of  creative  reward  systems 
to  achieve  acquisition  reform  initiatives.  They  define  an  acquisition  environment 
that  would  balance  responsibility  with  authority,  reduce  the  burden  of  mandatory 
procedures  and  specifications,  encourage  prudent  risk  management,  and  reduce 
reporting  requirements.  DoD  Directive  5000.1  specifies  that  incentives  shall  be 
applied  to  both  government  and  industry  to  achieve  the  objectives  of  cost  as  an 
independent  variable  (CAIV).  The  directive  also  states  that  award  programs  (both 
monetary  and  nonmonetary)  and  shared-savings  programs  shall  be  used  creatively 
to  encourage  cost-saving  ideas  for  all  phases  of  the  acquisition  life-cycle.  Incen¬ 
tive  programs  are  to  target  individuals  and  teams  in  both  government  and  industry, 
and  shall  stress  up-front  investments  to  minimize  production  and/or  operation  and 
support  costs,  where  applicable.6 

The  Office  of  Personnel  Management  (OPM)  clearly  favors  an  approach  that  links 
organizational  performance  to  appraisals  and  rewards.  Although  there  is  no  ex¬ 
plicit  statutory  authority  for  granting  awards,  Title  V,  United  States  Code 
(U.S.C.),  states  that  the  head  of  an  agency  may  “pay  a  cash  award  to  and  incur 
necessary  expense  for  the  honorary  recognition”  of  an  employee  who  makes  any 
of  several  forms  of  contribution,  and  grants  authority  to  pay  individual  cash 
awards  up  to  $10,000  annually.7  Additionally,  Title  V  authorizes  OPM,  by  regula¬ 
tion,  to  permit  agencies  to  “grant  employees  time  off  from  duty,  without  loss  of 
pay  or  charge  to  leave,  as  an  award.  ”8 

OPM  recently  issued  guidance  concerning  the  use  of  nonmonetary  and  cash- 
surrogate  awards  as  incentives.9  Specifically,  the  policy  lists  criteria  for  honorary 
awards  and  informal  recognition : 

♦  Honorary  awards  (formal  recognition) 

>-  cannot  convey  a  sense  of  monetary  value, 

►  must  have  lasting  trophy  value, 


4  For  more  information,  see  http://www.npr.gov/initiati/21cent/index.html. 

5  “Eyes  on  the  Prize,”  by  Howard  Risher,  Government  Executive,  September  1997. 

6  DoD  Directive  5000.1,  part  3,  paragraph  3.3.3.2,  March  1996. 

7  5  U.S.C.  4503. 

8  5  U.S.C.  4502(e). 

9  Using  “Nonmonetary  Items”  as  Incentive  Awards;  Using  “Cash  Surrogates”  to  Deliver 
Cash  Awards;  USOPM:  HRSS:  OERWP:  PMIAD,  January  1997. 
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>-  should  symbolize  the  employer-employee  relationship,  and 

>  must  be  appropriate  for  expenditure  of  public  sector  funds. 

♦  Awards  used  as  informal  recognition  must  have  a  nominal  value  ($100  or 
less)  and  be  appropriate  for  expenditure  of  public-sector  funds. 

A  special  case  has  been  made  for  U.S.  Savings  Bonds.  OPM  considers  them  to  be 
a  nonmonetary  item  because  they  are  a  federal  contract  that  must  be  purchased, 
and  no  value  limit  has  been  placed  on  their  use. 

Over  the  past  four  to  five  years,  notable  legislation  and  policy  have  recognized 
that  pay,  reward,  and  incentive  systems  should  be  better  aligned  with  organiza¬ 
tional  performance.  In  the  wake  of  the  1993  GPRA  goal-setting  and  performance 
measurement  requirements,  legislation  and  policy  are  recommending  broad  dis¬ 
cretion  to  design  leading-edge  reward  systems. 

Industry  Compensation  Trends 

We  examined  industry  trends  and  initiatives  relating  to  reward  and  performance 
management  strategies  and  their  effect  on  employee  motivation  and  performance. 
Generally,  as  a  result  of  downsizing,  re-engineering,  and  restructuring,  traditional 
organizational  hierarchies  are  disappearing,  organizations  are  becoming  flatter  and 
reorganizing  into  multi-disciplined,  cross-functional  teams.  Performance  man¬ 
agement  and  related  compensation  systems  are  being  redesigned  to  recognize  and 
reward  people  for  what  they  know  and  how  they  perform,  not  for  their  place  on  an 
organizational  chart — in  effect,  “paying  people,  not  jobs.”10  Moreover,  companies 
are  making  changes  in  order  to  link  compensation  more  closely  to  business  strat¬ 
egy  and  outcomes.11 

Figure  1-1  shows  how  pervasive  new  compensation  practices  are.12  This  informa¬ 
tion  reflects  the  responses  of  297  compensation  and  human  resources  profession¬ 
als  and  executives  (from  the  United  States  and  Canada)  attending  the  1996  Hay 
Compensation  Conference. 


10  “Paying  People,  Not  Jobs,  Reward  Systems  in  the  New  Logic  Corporation,”  Edward  E. 
Lawler  III,  Ph.D.,  ACA  News,  May  1 997. 

11  Work  in  the  21st  Century:  Implications  for  Compensation,  Karen  E.  May,  Human  Resource 
Solutions,  Orinda,  CA,  June  1997. 

12  Reprinted  with  permission  from  The  Hay  Report,  Compensation  and  Benefits  Strategies  for 
1998  and  Beyond,  Hay  Group,  Inc.  1997,  p.  3-1. 
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Figure  1-1.  Use  of  Innovative  Compensation  and  Incentive  Programs 
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Lump-sum  payments  (45  percent)  and  one-time  spot  awards  (37  percent)  are  the 
most  commonly  used  compensation  programs  among  respondents.13  These  types 
of  incentives  are  not  noted  as  strong  motivators  of  behavior  change,  but  they  are 
believed  to  be  appropriate  in  the  right  situation  and  with  other  elements  intro¬ 
duced  in  the  mix.  More  ambitious  plans  such  as  team-based  pay  and  pay  for  com¬ 
petencies  fall  near  the  bottom  of  the  list,  however,  more  companies  are 
considering  these  plans  because  they  can  be  tailored  to  reinforce  internal  change 
and  alter  individual  and  group  behaviors.  Substantial  growth  is  likely  in  these  ar¬ 
eas  as  companies  come  to  rely  on  the  development  of  competencies  and  organize 
more  into  teams. 

In  Figure  1-2,  the  order  changes  when  the  question,  “Has  the 
[compensation/innovative  incentive]  program  helped  improve  performance?”  is 
asked.  Team-based  pay,  gainsharing,  and  group  incentives  move  to  the  top  of  the 
list,  and  lump-sum  payments  and  special  one-time  “spot”  awards  move  down  in 
value.14  These  results  articulate  the  value  of  incentive  strategies  linked  to  team 
performance. 


13  1996  Hay  Fall  Conference  Survey,  The  Hay  Report,  Compensation  and  Benefits  Strategies 
for  1998  and  Beyond,  Hay  Group,  Inc.,  1997,  p.  3-1. 

14  Ibid.,  p.  3-3 


1-5 


Figure  1-2.  Performance  Improvement 
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Reward  and  recognition  programs  in  the  private  sector  are  being  structured  to 
support  team  effectiveness  and  to  provide  incentives  that  motivate  teams  to 
achieve  common  objectives.  This  is  particularly  relevant  to  DoD’s  PMT  environ¬ 
ment.  Teams  enable  employees  to  collectively  enhance  organizational  perform¬ 
ance  and  achieve  key  objectives.  Compensation  practices  such  as  performance 
bonuses,  team-based  rewards,  and  goalsharing  support  these  trends. 

The  Changing  Program  Management  Team  Environment 

In  the  past,  the  PMT  environment  was  characterized  by  ambiguities  and  conflicts 
in  the  roles  and  responsibilities  of  program  managers,  burdensome  oversight  and 
review  activities,  and  deficiencies  in  acquisition  qualifications  and  training.  Now, 
in  the  midst  of  downsizing,  rightsizing,  and  a  host  of  legislative  changes  and  ac¬ 
quisition  reform  initiatives  that  have  forced  dramatic  change  to  achieve  the  effi¬ 
ciency  required  by  fiscal  realities.  Program  Managers  (PMs)  are  implementing 
innovative  strategies  to  “do  more  with  less.”  PMTs  are  committed  to  eliminating 
unnecessary  regulation,  delegating  decision  authority  to  the  lowest  possible  or¬ 
ganizational  level  (empowerment),  using  commercial-off-the-shelf  (COTS)  prod¬ 
ucts  and  equipment,  and  implementing  best  commercial  practices  when  doing  so 
is  in  the  best  interest  of  the  warfighter,  their  military  service,  and  the  government. 
PMs  are  forming  partnerships  with  industry  and  paying  incentives  to  industry  in 
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the  form  of  cost-plus-award-fee  contracts,  to  encourage  the  development  and  ap¬ 
plication  of  cost  reduction  strategies  and  creative  ways  to  implement  them. 

Integrated  Product  Teams 

Integrated  Product  Teams  (IPTs)  have  been  established  to  optimize  design,  manu¬ 
facturing,  and  supportability  processes.  These  multidisciplinary  teams  bring  peo¬ 
ple  from  appropriate  functional  disciplines  together  to  identify  and  resolve  issues 
early  in  the  process  and  build  successful,  balanced  programs.  They  are  established 
to  enable  making  the  right  decisions  at  the  right  time  and  are  being  used  success¬ 
fully  by  industry  and  government  program  offices.  DoD  mandated  the  IPT  concept 
to  help  shift  from  “an  environment  of  regulation  and  enforcement  to  one  of  incen- 
tivized  performance,. ..  and  to  create  a  climate  of  reasoned,  well-informed  risk 
management  by  the  PM’s  and  Program  Executive  Officers  (PEOs).”15 

Multiple  Human  Resource  Management  Systems 

The  PMT  environment  represents  a  unique  study  of  team  dynamics  because  at  any 
one  time,  three  distinctly  different  human  resource  management  systems  are  at 
work  reporting,  appraising,  and  compensating  individual  performance — one  each 
for  government  civilians,  military,  and  SETA  contractors.  OPM  provides  policy 
for  individual  performance  management  (civilians  only)  but  very  little  in  terms  of 
team-based  performance  management.  Several  ongoing  personnel  demonstration 
projects,  with  the  authority  to  waive  Title  V  provisions,  are  introducing  perform¬ 
ance  and  compensation  systems  designed  to  be  more  responsive  and  productive. 
However,  participation  is  limited  to  civilians  only,  and  these  tests  do  not  evaluate 
team  performance  or  provide  significant  incentives  to  motivate  team  performance. 

The  8th  Quadrennial  Review  of  Military  Compensation  (QRMC)  concluded  that 
the  design  of  a  [reward]  system  to  motivate  the  workforce  of  the  future  would  be 
seriously  compromised  if  other  components  of  the  human  resource  management 
system  (compensation,  personnel  management,  and  organizational  design)  were 
not  taken  explicitly  into  account  and  given  due  consideration.16  Both  academics 
and  the  QRMC  research  agree  that  the  human  resource  systems  should  be  de¬ 
signed  to  be  responsive  to  strategic  and  organizational  changes  and  that  reward 
and  recognition  programs  should  be  structured  to  support  team  effectiveness, 
drive  performance,  and  reinforce  desired  behaviors.17  Managers  need  to  be  able  to 
use  the  human  resource  system  to  facilitate  and  reinforce  the  decisions  they  make 


15  Remarks  to  the  Industrial  College  of  the  Armed  Forces:  “The  Defense  Acquisition  Chal¬ 
lenge:  Technological  Supremacy  at  an  Affordable  Cost,”  Honorable  Paul  G.  Kaminski,  January  27, 
1995. 

16  Rewarding,  Organizing  and  Managing  People  in  the  2 1st  Century:  Time  for  a  Strategic 
Approach.  Report  of  the  8th  QRMC,  June  30, 1997. 

17  “Paying  People,  Not  Jobs,  Reward  Systems  in  the  New  Logic  Corporation,”  Edward  E. 
Lawler  ID,  Ph.D,  ACA  News,  May  1997. 
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about  how  to  organize,  manage,  and  reward  people,  and  to  select  the  policies  and 
practices  that  target  desired  behaviors. 

In  order  to  motivate  the  workforce  and  achieve  performance  results,  it  is  necessary 
to  change  from  an  entitlement  culture  (entitled  to  rewards,  uninspired  about  earn¬ 
ing  them  because  they  are  based  on  equity  rather  than  on  performance)  to  a  per¬ 
formance-based  culture  and  to  inspire  people  with  the  energy  of  knowing  that  the 
results  of  their  accomplishments  are  noticed  and  rewarded. 

Research  Approach 

ACAT I  programs  are  subject  to  numerous  destabilizers  derived  from  outside  the 
acquisition  program  (e.g.,  changing  user  requirements,  lack  of  budget/funding  sta¬ 
bility,  schedule  or  quantity  changes)  that  have  an  impact  on  cost,  schedule,  and 
performance  outcomes.  To  test  the  hypothesis  that  enhanced  incentives  will  have  a 
positive  impact  on  program  success,  it  was  necessary  that  we  approach  the  task 
from  two  perspectives: 

♦  Develop  a  reward  framework  that  ensures  incentives  are  linked  to  factors 
within  the  control  of  the  PMT. 

♦  Design  a  statistical  test,  that  controls  for  external  destabalizers,  to  deter¬ 
mine  whether  workforce  incentives  contribute  to  program  success,  as 
measured  by  cost,  schedule,  and  performance  parameters. 

These  two  sub-tasks  reinforce  the  foundation  of  our  research  approach,  namely  that 
incentives  must  have  good  line  of  sight  to  performance  and  the  incentive  system 
must  relate  to  critical  processes  or  outcomes  that  the  PMT  can  influence.  Therefore, 
sub-task  one  will  develop  a  performance  measurement  framework  and  introduce 
incentives  linked  to  factors  within  the  PMT’s  control.  The  statistical  test  design, 
sub-task  two,  relies  on  metrics  that  capture  organizational  outcomes  in  terms  of 
cost,  schedule,  and  performance.  These  measures  will  attempt  to  determine  whether 
linking  incentives  to  performance  measures  that  the  team  can  control  (within  good 
line  of  sight)  will  have  a  positive  impact  on  program  success  and  allow  for  the 
comparison  of  test  and  control  groups. 

Figure  1-3  illustrates  the  operational  environment  in  which  the  pilot  sites  will  oper¬ 
ate  and  which  the  test  design  must  accommodate.  The  incentives  system  will  be 
linked  to  processes  and  factors  within  the  teams’  control.  The  test  outcomes  will 
measure  the  effect  on  cost,  schedule,  and  performance,  which  should  be  directly  or 
indirectly  affected  by  incentives.  The  test  design  must  control  for  the  effects  of 
external  factors  beyond  the  team’s  control  that  affect  cost,  schedule,  or  perform¬ 
ance. 
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Figure  1-3.  Operational  Environment 


Developing  the  Incentive  System 

For  the  PMT  environment,  incentives  should  be  structured  to  support  team  effec¬ 
tiveness  and  to  motivate  teams  to  achieve  common  objectives.  Although  work¬ 
place  teams  are  considered  one  of  the  most  sweeping  management  innovations  of 
the  past  decade,  many  organizations  have  failed  to  change  their  compensation  and 
reward  policies  to  match  or  motivate  the  new  team  cultures.  The  challenge  is  to 
design  a  system  that  develops  individuals,  motivates  the  right  performance,  and 
ties  rewards  to  the  group’s  accomplishments.  To  meet  this  challenge,  rewards 
must  be  meaningful,  there  must  be  good  line  of  sight  between  what  is  measured 
and  what  employees  can  control,  and  rewards  must  be  linked  to  a  credible  per¬ 
formance  management  system  to  reinforce  desired  behavior  and  to  motivate  teams 
to  achieve  desired  objectives  (i.e.,  drive  performance). 

Designing  the  Test 

The  test  of  any  reward  system  is  whether  it  encourages  behavior  that  improves 
performance  and  then  rewards  the  results  of  that  behavior.  The  test  we  have  de¬ 
signed  will  measure  the  impact  of  workforce  incentives  on  program  success  while 
neutralizing  the  effects  of  external  factors  known  to  influence  the  program.  Out¬ 
comes  must  measure  the  effect  of  incentives  on  program  cost,  schedule,  and  per¬ 
formance  in  an  effort  to  capture  program  success.  We  will  compare  test  programs 
that  use  enhanced  incentives  with  control  programs  that  do  not  to  determine 
whether  incentives  can  explain  differences  in  program  performance. 

Chapter  2,  Functional  Test  Description,  explains  how  we  formulated  the  concepts 
on  which  the  test  is  based,  describes  what  we  will  do  in  terms  of  test  fundamen¬ 
tals,  and  then  discusses  how  we  will  control  for  extraneous  variables.  Chapter  3, 
Detailed  Test  Design,  illustrates  the  mechanics  of  measuring  response  and 
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determining  the  sample  size  (how  many  pilot  sites  should  participate).  Chapter  4, 
Implementation,  presents  an  implementation  plan  in  two  stages:  stage  one  to  es¬ 
tablish  a  team-based  performance  measurement  system,  and  stage  two  to  link  in¬ 
centives  to  stretch  targets  that  relate  to  strategic  objectives.  A  set  of 
implementation  prerequisites  for  each  phase  is  outlined.  These  were  developed  on 
the  basis  of  research  findings,  program  office  interviews,  and  the  Acquisition 
Workforce  Enhanced  Incentives  Survey.18  The  survey  was  conducted  to  support 
research  and  interview  findings  that  rewards  have  a  large  potential  to  influence 
program  success.  The  prerequisites  provide  the  conditions  necessary  to  proceed 
through  the  two  implementation  stages.  The  report  concludes  with  Final  Thoughts 
in  Chapter  5. 


18  LMI  Report  AQ703T1 ,  Acquisition  Workforce  Enhanced  Incentive’s  Survey  Analysis, 
Philippe  A.  Lussier,  Peggy  A.  Miller,  Albert  H.  Schroetel,  June  1998. 
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Chapter  2 

Functional  Test  Description 


Objective 

The  objective  of  the  test  is  to  determine  whether  enhanced  incentives  can  contrib¬ 
ute  to  program  success.  The  hypothesis  it  examines  is  that  incentives  will  have  a 
positive  impact  on  cost,  schedule,  and  performance. 

Key  Steps 

The  incentives  system  must  relate  to  critical  processes  or  outcomes  that  the  PMT 
can  influence.  The  test  is  structured  to  determine  whether  or  not  these  processes  or 
outcomes  can  have  a  positive  impact  on  cost,  schedule,  and  performance.  The  key 
steps  in  designing  the  test  included 

♦  establishing  the  baseline  from  which  to  measure  the  effects  of  incentives, 

♦  implementing  a  performance  measurement  system, 

♦  determining  how  to  control  for  external  and  internal  factors  of  non-interest 
known  to  influence  program  success, 

♦  introducing  enhanced  incentives  linked  to  the  performance  measurement 
system, 

♦  identifying  a  statistical  methodology  for  measuring  program  success, 

♦  determining  how  many  teams  should  participate  in  order  to  collect  suffi¬ 
cient  data,  and 

♦  collecting  data,  performing  analysis,  and  interpreting  the  results. 

The  first  four  steps  relate  to  the  functional  design  of  the  test  and  are  covered  in 
this  chapter.  The  remaining  steps,  which  require  an  empirical  foundation,  are  dis¬ 
cussed  in  Chapter  3,  Detailed  Test  Design. 
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Formulating  Test  Concepts 


Integral  to  developing  the  incentive  system  and  the  corresponding  test  design  were 
the  inputs  and  insights  gained  from  program  team  members  and  other  stakehold¬ 
ers.  Their  depth  of  knowledge  and  their  cooperation  have  been  invaluable  in  for¬ 
mulating  test  concepts  and  building  a  knowledge  base. 

This  study  began  with  research  of  academic  literature  and  practitioner  experiences 
pertaining  to  rewards  and  recognition.  We  also  extensively  reviewed  government 
and  industry  initiatives  related  to  performance  management  strategies  and  their 
effect  on  employee  motivation  and  performance.  We  met  with  subject  matter  ex¬ 
perts  from  DoD,  OPM,  and  the  private  sector,  including  civilian  personnel  spe¬ 
cialists,  military  compensation  managers,  senior  acquisition  leaders,  management 
consultants,  and  technology  integration  professionals. 

We  conducted  interviews  with  PEOs,  PMs,  Deputy  PMs,  and  team  members  at  all 
levels  in  the  program  offices.  Additionally,  we  conducted  a  survey  of  senior  level 
acquisition  managers  and  program  office  personnel  to  determine  what  rewards  are 
most  valued  by  the  workforce,  to  identify  critical  program  management  processes 
and  their  relationship  to  achieving  cost,  schedule  and  performance  goals,  and  to 
determine  how  effective  current  performance  management  practices  are.  Highlights 
of  the  interviews  and  survey  results  are  captured  in  the  following  sections.  A  de¬ 
tailed  discussion  of  survey  results  and  the  survey  instrument  can  be  found  in  Ap¬ 
pendix  A,  with  its  Annex  1  and  Annex  2. 

Interview  Findings 

There  was  a  high  degree  of  interest  in  incentives  in  all  of  the  organizations  we 
visited.  It  was  generally  agreed  that  rewards  and  recognition  have  a  significant 
potential  to  influence  program  success.  There  is  not  always  a  good  line  of  sight, 
however,  to  aggregate  measures  of  program  performance  such  as  cost,  schedule, 
and  technical  performance.  Program  managers  acknowledged  that  the  Acquisition 
Program  Baseline  (APB),  which  captures  the  program’s  key  cost,  schedule,  and 
performance  parameters,  is  their  “contract  with  the  government,”  but  most  team 
members  agreed  that  the  metrics  used  to  measure  team  and  program  performance 
do  not  reflect  the  program’s  critical  aspects. 

If  rewards  are  to  be  used  successfully  to  motivate  people  and  drive  performance  in 
this  environment,  program  team  members  at  all  levels  told  us  the  following: 

♦  The  team  succeeds  or  fails  collectively,  and  should  be  rewarded  collec¬ 
tively. 

♦  All  team  members  should  be  eligible  to  participate  in  the  rewards  program, 
including  civilians,  military,  and  SETA  support  contractors. 
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♦  Rewards  should  be  team-based,  with  an  opportunity  for  the  team  to  recog¬ 
nize  individuals. 

♦  Either  cash  awards  or  savings  bonds  must  be  available  to  all  team  mem¬ 
bers  (as  a  stimulus  for  the  shift  to  a  performance-based  culture). 

♦  Rewards  must  be  equitable. 

♦  A  credible  performance  management  system  (to  link  incentives  to)  must 
be  established. 

Survey  Results 

The  survey  helped  us  to  quantify  what  rewards  are  valued  by  the  workforce  and  to 
determine  whether  enhanced  incentives  can  contribute  to  program  success.  It  was 
important  for  us  to  analyze  the  feasibility  of  using  program  cost,  schedule,  and 
performance  goals  to  evaluate  acquisition  personnel  performance.  The  survey 
provided  empirical  evidence  to  support  interview  findings  that  rewards  have  a 
large  potential  to  influence  program  success,  and  that  an  incentive  program  linked 
to  a  credible  performance  measurement  system  could  provide  a  model  for  change 
in  the  acquisition  program  environment.  The  survey  indicated  the  following: 

♦  Current  performance  management  systems  lack  credibility. 

♦  Team  performance  is  managed  less  effectively  than  individual  perform¬ 
ance. 

♦  External  factors  have  a  large  influence  on  program  success. 

♦  Both  civilians  and  military  place  a  high  value  on  monetary  awards. 

We  asked  participants  to  what  extent  they  personally  valued  various  rewards,  to 
determine  what  types  of  rewards  would  provide  the  greatest  motivational  value. 
Table  2-1  presents  the  overall  results,  ranked  from  highest  to  lowest  value. 

Results  revealed  that  base  pay  increases,  cash  awards,  and  outstanding  perform¬ 
ance  ratings  were  invariably  ranked  the  top  three,  while  productivity  upgrades, 
administrative  support  (considered  enablers  and  not  rewards),  and  gift  certificates 
ranked  consistently  as  the  bottom  three. 
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Table  2-1.  Rewards,  Ranked  by  Mean  Value 


Rank 

Reward 

1 

Base  pay  increase 

2 

Cash  award 

3 

Outstanding  rating 

4 

Paid  time  off 

5 

Savings  bonds 

6 

Education  &  training 

7 

Assignment  preference 

8 

Flexible  work  hours  &  place 

9 

Influence  in  goal  setting 

10 

Assignment  of  high-status  tasks 

11 

Informal  recognition 

12 

Formal  recognition 

13 

Tuition  refunds 

14 

Unused  leave  sell-back 

15 

Productivity  upgrades 

16 

Administrative  support 

17 

Gift  certificates 

Test  Fundamentals 

This  section  describes  the  fundamentals  of  conducting  the  acquisition  workforce 
enhanced  incentives  test.  The  test  must  consider  several  key  questions: 

♦  What  types  of  culture  and  organizational  strategies  should  the  incentives 
system  support? 

♦  Can  enhanced  incentives  improve  the  likelihood  of  program  success? 

♦  How  much  can  program  success  be  influenced  by  incentives  for  govern¬ 
ment  program  office  personnel? 

♦  Can  larger  incentives  result  in  higher  performance? 

♦  What  types  of  incentives  (monetary  or  nonmonetary)  will  work  best  in  this 
environment? 

♦  How  can  the  incentives  system  be  implemented  without  significant  in¬ 
creases  in  reporting  and  administrative  requirements? 
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The  test  will  incorporate  key  aspects  of  classical  test  design,  modified  for  practi¬ 
cality  and  real-world  considerations.  The  first  step  is  establishing  a  baseline  from 
which  to  measure  the  effect  of  the  incentives.  Second,  since  stakeholders  told  us 
that  current  appraisal  systems  are  not  appropriate  for  linking  incentives  to  per¬ 
formance,  implementing  a  credible,  team-based  performance  measurement  system 
to  which  incentives  can  be  linked  is  essential.  Third,  because  time  will  have 
elapsed  while  the  performance  measurement  system  is  being  established  and  the 
system  itself  will  introduce  some  change,  the  baseline  will  have  to  be  reestab¬ 
lished  before  incentives  can  be  introduced.  The  fourth  step  is  to  introduce  incen¬ 
tives  designed  to  drive  performance  in  support  of  the  objectives  established  for  the 
performance  measurement  system.  The  last  step  is  collecting  the  data  and  assess¬ 
ing  the  results. 

The  pilot  incentive  programs  will  be  continuously  evaluated  and  the  incentives 
systems  adjusted  accordingly.  By  experimenting  and  incrementally  gaining  expe¬ 
rience,  lessons-leamed  and  successful  practices  can  be  leveraged  to  expand  the 
program  efficiently  and  wisely. 

Establishing  the  Baseline 

Baseline  data  will  be  collected  for  test  and  control  groups  before  the  test  is  initi¬ 
ated.  The  data  collection  efforts  will  rely  heavily  on  information  already  assimi¬ 
lated  by  major  defense  acquisition  programs.  Measurements  will  be  taken  of  the 
variables  that  we  expect  incentives  to  affect.  Those  variables  in  aggregate,  will 
reflect  some  measure  of  program  success  and  include  the  program’ s  cost,  sched¬ 
ule,  and  performance  characteristics.  The  APB  will  be  used  to  baseline  the  pro¬ 
gram  and  to  capture  program  measures  for  cost,  schedule,  and  performance.  The 
methodology  for  calculating  these  measures  is  discussed  further  in  the  Measuring 
Response  section  of  Chapter  3. 

In  addition  to  operational  outcomes  such  as  cost,  schedule,  and  performance,  it  is 
our  hypothesis  that  incentives  used  properly  will  affect  individual  and  team  mo¬ 
rale,  motivation,  and  job  satisfaction,  in  turn  leading  to  improved  performance1. 
These  behavioral  outcomes  are  hard  to  capture  or  quantify,  but  they  will  reflect 
how  well  organizational  strategy  and  desired  outcomes  are  communicated,  and 
whether  employees  sense  leadership  commitment  to  reform  initiatives.  To  capture 
these  effects,  a  climate  survey  will  be  conducted  prior  to  test  initiation  and  peri¬ 
odically  thereafter  to  determine  if  improvements  have  been  made  in  terms  of  job 
satisfaction  and  motivation.  The  survey  used  for  the  stakeholder  involvement  por¬ 
tion  of  this  study  (included  in  Annex  1  of  Appendix  A)  will  be  modified  to  focus 
more  specifically  on  individual  and  team  performance. 


1  “Profit  Sharing  and  Productivity,”  Martin  L.  Weitzman  and  Douglas  L.  Kruse,  Paying  for 
Productivity,  A  Look  at  the  Evidence,  Alan  S.  Blinder,  Editor,  Section  3,  p.  139. 
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Implementing  the  Performance  Measurement  System 

Our  research  has  shown  that  before  incentives  are  introduced,  a  credible  team- 
based  performance  measurement  system  must  be  implemented.  The  survey  pro¬ 
vided  evidence  that  existing  performance  appraisal  systems  are  not  a  credible 
measure  of  job  performance  and  do  not  effectively  measure  team  performance. 

After  an  extensive  review  of  government  and  industry  initiatives  with  regard  to 
performance  management  strategies,  we  have  determined  that  the  balanced  score- 
card  (Figure  2-1)  represents  the  leading  edge  of  management  thinking  over  the 
past  five  years.  The  Gartner  Group  estimates  that  40  percent  of  Fortune  1000 
Companies  will  implement  a  new  management  system — the  Balanced  Score- 
card — by  the  year  2000. 

The  Balanced  Scorecard  is  a  performance  measurement  system  that  provides  or¬ 
ganizations  a  framework  for  translating  mission  and  strategy  into  objectives  and 
measures,  as  shown  in  Figure  2-1. 2  It  is  organized  across  four  balanced  perspec¬ 
tives:  financial,  customer,  internal  business  processes,  and  learning  and  growth. 
By  clearly  defining  the  outcomes  an  organization  desires  and  the  drivers  of  those 
outcomes,  senior  managers  can  channel  the  energies,  abilities,  and  specific  knowl¬ 
edge  of  people  throughout  the  organization  toward  achieving  the  long-term  goals. 

Figure  2-1.  The  Balanced  Scorecard 


2  “The  Balanced  Scorecard — Measures  that  Drive  Performance,”  Robert  S.  Kaplan  and  David 
P.  Norton,  Harvard  Business  Review,  January-February  1992. 


2-6 


Functional  Test  Description 


The  Balanced  Scorecard  is  already  being  widely  used  as  a  strategic  management 
system.3  Mobil  Oil’s  U.S.  Marketing  &  Refining  Division  (USM&R)  developed 
its  scorecard  during  1994  to  link  the  entire  organization  (18  business  units  and  14 
service  units)  and  drive  alignment  from  top  to  bottom.  Each  business  and  service 
unit  now  has  a  scorecard  aligned  to  the  division  scorecard  (and  thus  its  strategy) 
through  a  carefully  chosen  core  of  objectives  and  measures.  Since  1993,  Mobil 
Oil  (USM&R)  has  gone  from  number  seven  in  profitability  in  the  oil  industry  to 
number  one  for  an  unprecedented  three  consecutive  years.  It  attributes  this  turn¬ 
around  to  the  scorecard.4 

CIGNA  Property  and  Casualty  (P&C)  developed  its  first  balanced  scorecard  to 
create  a  new  vision  for  itself  as  an  underwriting  specialist.  But  once  CIGNA 
started  using  it,  the  scorecard  allowed  the  chief  executive  officer  (CEO)  and  the 
senior  management  team  not  only  to  introduce  a  new  strategy  for  the  organization, 
but  to  overhaul  the  company’s  entire  management  system.  The  CEO  was  able  to 
transform  the  company  so  that  everyone  could  focus  on  achieving  long-term  stra¬ 
tegic  objectives.  Since  1993,  the  company  has  gone  from  $275  million  in  the  red 
to  over  $100  million  in  profit,  and  its  stock  price  has  risen  from  $62  to  $172  per 
share.5 

Both  the  U.S.  Department  of  Transportation  and  the  U.S.  Department  of  the 
Treasury  have  utilized  a  balanced  scorecard  to  reengineer  the  federal  procurement 
oversight  process.  Instead  of  reviewing  contract  files  for  compliance,  these  agen¬ 
cies  use  a  results-oriented,  performance  measurement  process  that  focuses  on 
customer  satisfaction,  employee  empowerment,  and  management  assessments  and 
emphasizes  prevention  rather  than  detection.  Their  cross-agency  effort  to  reduce 
costs  and  improve  results  won  them  the  International  Benchmarking  Clearing¬ 
house  (IBC)  Gold  Award  in  Applied  Research  and  the  Vice  Presidential  Hammer 
Award.6  These  agencies  are  considered  a  benchmark  in  terms  of  compliance  with 
GPRA. 

The  Information  Technology  Management  Reform  Act  requires  the  federal  agen¬ 
cies,  which  spend  over  $25  billion  annually  on  information  technology  (IT)  prod¬ 
ucts  and  services,  to  measure  the  contribution  of  their  IT  investments  to  their 


3  “Using  the  Balanced  Scorecard  as  a  Strategic  Management  System,”  Robert  S.  Kaplan  and 
David  P.  Norton,  Harvard  Business  Review,  January-February  1996. 

4  Mr.  Edward  T.  Lewis,  Jr.,  Supervisor,  Strategic  Planning,  U.S.  Marketing  &  Refining  Divi¬ 
sion,  Mobil  Oil  Corporation. 

5  Thomas  M.  Valerio,  Senior  Vice  President,  Transformation  Officer,  CIGNA  P&C. 

6  IBC  is  a  service  of  the  American  Productivity  and  Quality  Center  (APQC).  IBC  was  de¬ 
signed  to  help  managers  find  and  adopt  best  practices.  Clearinghouse  members  include  hundreds  of 
companies,  government  agencies,  healthcare  providers,  and  educational  institutions.  For  more  in¬ 
formation,  see  www.apqc.org. 
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mission  results.7  The  General  Services  Administration  used  a  balanced  scorecard 
to  develop  and  use  IT  performance  measures. 

Further  information  regarding  the  Balanced  Scorecard  can  be  found  in  Appendix 
B.  How  the  scorecard  will  be  implemented  in  the  program  management  offices  is 
covered  in  Chapter  4. 

Reestablishing  the  Baseline 

It  is  our  hypothesis  that  a  new  performance  measurement  system  will  have  its  own 
measurable  impact  on  achieving  cost,  schedule  and  performance  goals  established 
for  major  defense  acquisition  programs.  Therefore,  before  incentives  are  intro¬ 
duced,  the  baseline  must  be  reestablished.  The  process  of  implementing  the  bal¬ 
anced  scorecard  and  observing  the  strategic  measures  to  determine  whether  they 
are  capturing  and  communicating  the  proper  relationships  will  take  approximately 
9  to  12  months.  At  the  end  of  this  period,  when  employees  and  management  have 
confidence  that  the  system  is  communicating  and  measuring  performance  as  in¬ 
tended,  a  new  baseline  will  be  established.  The  APB  and  the  climate  survey  will 
be  the  source  material  for  establishing  baseline  response  measures.  These  meas¬ 
ures  are  covered  in  detail  in  Chapter  3. 

Introducing  Incentives 

As  senior  managers  continue  to  realize  the  need  to  align  employees  with  the  goals 
of  the  organization,  they  are  discovering  that  goal-setting  and  training  are  insuffi¬ 
cient  in  and  of  themselves.  What  is  needed  is  a  system  that  aligns  the  reward  strat¬ 
egy  with  the  goals  of  the  organization  and  in  doing  so,  aligns  the  employees  with 
the  organization.8 

An  incentive  system  should  reward  performance  on  the  basis  of  measures  within 
the  team’s  control.  Those  processes  that  have  the  greatest  effect  on  program  suc¬ 
cess  and  that  are  least  influenced  by  external  factors  become  the  best  candidates 
for  linking  performance  measures  and  rewinds. 

A  rewards  program  in  which  pay  is  linked  to  the  key  elements  of  the  scorecard 
provides  a  way  to  reinforce  team  efforts  to  support  the  achievement  of  strategic 
program  objectives.  Incentives,  particularly  monetary,  will  change  employee  be¬ 
havior,  and  it  is  important  that  the  measures  be  used  accurately  to  reflect  desired 
outcomes  and  that  process  relationships  are  understood.  Employees  and  manage¬ 
ment  should  feel  comfortable  with,  and  have  confidence  in,  the  objectives, 
measures,  and  targets  represented  in  the  scorecard.  The  organization  must  be 


7  Patrick  T.  Plunkett,  Program  Manager  for  Information  Technology  Performance  Measure¬ 
ment,  Genera]  Services  Administration. 

8  Designing  and  Managing  an  Organization-Wide  Incentive  Pay  System ,  William  B. 
Abernathy,  Ph.D,  Abernathy  and  Associates. 
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confident  that  the  behaviors  produced  by  incentives  will  support  their  strategic 
objectives. 

Assessing  the  Results 

We  anticipate  that  the  incentives  will  take  some  time  to  measurably  change  a  pro¬ 
gram’ s  bottom  line  in  terms  of  cost,  schedule,  and  performance.  In  practice,  per¬ 
sonnel  demonstration  projects  are  authorized  by  legislation  for  five-year  periods, 
since  interventions  that  change  the  personnel  management  system  require  time  for 
observing  the  effect  on  organizational  outcomes. 

In  many  cases,  strategic  measures  that  employees  may  have  a  good  line  of  sight  to 
and  that  will  be  reflected  in  the  scorecard  will  have  an  indirect  relationship  with 
cost,  schedule,  and  performance.  For  example,  a  Defense  Reform  Initiative  goal  is 
to  decrease  paper  transactions  by  50  percent  through  the  use  of  electronic  com¬ 
merce/data  interchange.  While  achieving  this  objective  may  be  worthy  of  an  in¬ 
centive  for  reaching  some  stretch  goal  (provided  there  is  a  good  measure  of 
“paper”  transactions),  it  would  take  some  time  to  observe  any  resultant  effect  on 
program  success. 

Not  all  program  offices  that  implement  the  scorecard  will  initially  participate  in 
enhanced  incentives,  particularly  the  authority  to  pay  cash  awards.  A  set  of  pro¬ 
gram  offices  will  be  reserved  as  control  groups,  with  no  enhancement  to  the  cur¬ 
rent  incentive  system,  to  determine  whether  statistical  evidence  supports  the 
hypothesis  that  enhanced  incentives  can  have  a  positive  impact  on  program  suc¬ 
cess.  Results  will  be  assessed  using  measures  from  the  APB  and  a  climate  survey. 

Controlling  for  Extraneous  Variables 

The  effect  of  extraneous  variables  must  be  controlled  for  in  order  to  isolate  the 
effects  of  the  factor  of  interest,  incentives.  Expected  outcomes  will  be  measured  in 
terms  of  cost,  schedule,  and  technical  performance.  If  a  large  enough  number  of 
test  and  control  groups  could  be  selected,  the  variables  of  non-interest  would 
equally  affect  the  outcomes  of  both  groups  and  in  fact  would  be  controlled  for  by 
randomness,  or  the  law  of  averages.  The  section  titled  Determining  Sample  Size, 
in  Chapter  3,  provides  the  statistical  rationale  for  choosing  the  number  of  test  and 
control  groups  to  participate  in  the  test. 

In  experimental  design,  extraneous  variables  are  controlled  by  blocking  the  ex¬ 
perimental  units.9  “Blocking”  refers  to  sorting  the  experimental  units,  in  this  case 
program  offices,  into  homogeneous  groups.  The  treatment  (incentives)  is  then  as¬ 
signed  at  random  within  the  blocks.  In  order  to  apply  this  technique  in  the  PMT 


9  “Randomized  Block  Designs,”  John  Neter  and  William  Wasserman,  Applied  Linear  Statisti¬ 
cal  Models,  1974,  p.  722. 
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environment,  we  must  consider  two  sets  of  extraneous  variables:  internal  and  ex¬ 
ternal. 

Internal  Factors 

Internal  factors  are  those  that  characterize  the  acquisition  program  itself.  A  RAND 
study  demonstrated  that  program  size  (in  terms  of  cost),  maturity,  phase,  weapon 
system  type,  military  service,  and  management  complexity  are  the  most  significant 
internal  influences  on  cost  growth.10  Of  these,  program  size,  maturity,  and  phase 
had  the  greatest  impact  on  cost  growth.  For  example,  small  programs  have  higher 
cost  growth,  possibly  as  a  result  of  proportionally  higher  research  and  develop¬ 
ment  costs  that  exhibit  higher  cost  variances  through  engineering,  manufacturing, 
and  development  (EMD)  or  production.  Older  programs  or  programs  with  longer 
life-cycles  on  average  reflect  higher  cost  growth  as  problems  accumulate  over 
time.  Development  cost  growth  is  higher  than  production  cost  growth  because  of 
the  technical  difficulties  reflected  in  research,  development,  test,  and  evaluation 
(RDT&E)  costs. 

Extraneous  factors  that  are  internal  and  are  characterized  by  program  demograph¬ 
ics  will  be  controlled  for  by  selecting  control  groups  of  similar  size,  phase,  and 
weapon  system  type  to  each  test  group.  In  this  manner,  the  blocking  technique  de¬ 
scribed  above  can  be  applied. 

External  Factors 

External  factors  are  derived  from  outside  the  acquisition  program  but  have  an  im¬ 
pact  on  program  cost,  schedule,  and  performance  outcome.  Examples  of  external 
factors  include 

♦  changing  user  requirements; 

♦  program  funding; 

♦  the  Planning,  Programming,  and  Budgeting  System; 

♦  Congress;  and 

♦  the  Office  of  the  Secretary  of  Defense. 

External  factors  will  be  controlled  for  using  an  arbiter  process.  The  arbiter  will 
determine  when  cost,  schedule,  or  performance  variance  is  the  result  of  an  exter¬ 
nal  influence.  A  variance  attributed  to  an  external  factor  will  be  discounted  before 
the  outcome  for  both  test  and  control  groups  is  measured.  The  arbiter  could  be  at 


10  An  Analysis  of  Weapon  System  Cost  Growth,  Drezner  et  al.,  RAND-Project  Air  Force, 
1993. 
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the  PEO,  Component  Acquisition  Executive  (CAE),  or  Overarching  Integrated 
Product  Team  level. 

An  example  of  an  external  influence  that  would  be  accounted  for  by  the  arbiter 
process  is  a  quantity  change.  Quantity  changes  may  result  from  the  authorization 
and  appropriation  process  or  changing  user  requirements.  In  these  cases,  as  deter¬ 
mined  by  an  arbiter,  costs  incurred  as  a  result  of  the  change  in  programmed  quan¬ 
tity  would  be  used  to  adjust  current  cost  estimates.  For  quantity  changes 
determined  to  be  due  to  external  influences,  the  current  cost  estimate  would  be 
adjusted  to  reflect  the  program  as  still  procuring  the  baseline  quantity.  In  some 
cases,  quantity  reductions  result  from  cost  overruns,  and  decisions  to  reduce 
quantity  are  made  to  keep  the  program  within  budget.  Since  such  quantity  changes 
are  not  the  result  of  external  influences,  they  should  not  be  discounted. 

Summary 

Stakeholder  input  indicates  that  current  appraisal  systems  lack  credibility  and  do 
not  support  team-based  performance  management.  With  regard  to  reward  and  rec¬ 
ognition,  both  military  and  civilian  respondents  consistently  placed  higher  value 
on  monetary  rewards  than  on  nonmonetary  rewards.  The  balanced  scorecard  was 
found  to  be  an  effective  strategic  management  tool  used  by  both  the  private  and 
public  sectors. 

The  test  will  consist  of  establishing  the  baseline,  implementing  the  performance 
management  system,  reestablishing  the  baseline,  introducing  the  incentives,  con¬ 
trolling  for  extraneous  variables,  and  assessing  the  results. 

The  following  chapter  provides  details  on  how  program  success  will  be  measured, 
states  how  many  program  offices  should  participate  and  why,  and  concludes  with 
test  recommendations. 
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Chapter  3 

Detailed  Test  Design 


Measuring  Response 

DoD  directives  require  every  acquisition  program  to  establish  goals  for  the  mini¬ 
mum  number  of  cost,  schedule,  and  performance  parameters  that  describe  the  pro¬ 
gram.1  Each  parameter  includes  both  an  objective  and  a  threshold  value. 

Threshold  values  are  individually  set  for  each  program  on  the  basis  of  its  charac¬ 
teristics  (e.g.,  maturity,  risk,  etc.).  If  the  threshold  values  are  not  specified,  the 
threshold  value  for  performance  will  be  equal  to  the  objective  value.  The  thresh¬ 
old  value  for  schedule  will  be  the  objective  value  plus  six  months.  The  threshold 
value  for  cost  will  be  the  objective  value  plus  10  percent.  Program  objective  and 
threshold  values  for  cost,  schedule,  and  performance  are  captured  in  the  APB. 

Test  outcomes  will  measure  the  effects  of  the  intervention  (incentives)  on  pro¬ 
gram  success.  Outcomes  for  test  programs  will  be  compared  to  outcomes  for  con¬ 
trol  programs  to  determine  whether  enhanced  incentives  can  explain  the 
differences  in  program  performance.  Outcomes  for  both  the  test  and  control 
groups  will  primarily  measure  cost  and  schedule  variances  from  the  APB.  Per¬ 
formance  breaches  will  be  recorded  in  both  test  and  control  groups,  however, 
there  is  usually  insufficient  variance  in  technical  performance  parameters  to  ex¬ 
pect  that  the  response  due  to  the  introduction  of  enhanced  incentives  would  be 
measurable.  Research  has  shown  that  acquisition  programs  usually  meet  their 
systems’  technical  performance  objectives.2  It  is  recommended  that  no  payment  of 
bonuses  be  made,  regardless  of  other  cost  or  schedule  measures,  unless  the  pro¬ 
gram  is  meeting  or  has  met  all  of  the  key  performance  parameter  thresholds  in  the 
APB. 

The  Acquisition  Program  Baseline 

The  APB  is  used  to  baseline  the  program  and  to  capture  program  measures  for 
cost,  schedule,  and  performance.  We  intend  to  avoid  introducing  new  reporting 
requirements  for  the  purpose  of  conducting  the  incentives  test.  The  APB  is  recog¬ 
nized  as  the  Program  Manager’s  contract  with  decision  authorities.  APB  devia¬ 
tions  are  reported  annually  unless  a  program  is  in  breach,  in  which  case  quarterly 
reporting  can  be  mandated.  ACAT IA  programs  report  APB  deviations  in  quar¬ 
terly  major  automated  information  system  (MAIS)  reports  to  the  decision 


1  DoD  5000.2-R. 

2  An  Analysis  of  Weapon  System  Cost  Growth,  Drezner  et  al.,  RAND-Project  Air  Force,  1993. 
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authority.3  MAIS  reports  are  not  currently  required  by  statute  and  are  not  usually 
sent  to  Congress.  ACAT IC/D4  programs  report  APB  deviations  to  Congress  in  the 
form  of  Selected  Acquisition  Reports  (SARs).  The  SARs  are  a  mechanism  for  re¬ 
cording  and  tracking  program  cost,  schedule,  and  performance  variations.  We  will 
use  the  information  already  captured  in  the  SARs  for  the  purpose  of  this  incen¬ 
tives  test. 

Table  3-1  is  a  generic  SAR  planning  matrix.  Within  a  phase  or  baseline,  three  sets 
of  numbers  or  dates  are  tracked  with  respect  to  cost,  schedule,  and  performance. 
When  a  program  first  enters  a  phase,  the  values  in  all  three  columns  are  the  same. 
The  original  Planning  Estimate  (column  one)  remains  the  same  throughout  the 
phase.  A  submission  to  revise  the  APB  as  a  result  of  cost  breach  or  schedule  de¬ 
viation  results  in  adjustments  to  the  Approved  Program  (column  two).  The  Cur¬ 
rent  Estimate  (column  three)  is  updated  as  the  phase  progresses.  Response 
measures  for  the  enhanced  incentive  test  will  use  the  Current  Estimate  as  a  base¬ 
line  for  both  the  test  and  control  groups.  This  baseline  will  be  established  when 
the  test  begins,  and  all  measurements  will  be  taken  from  it. 

Table  3-1.  SAR  Planning  Matrix 


Item  tracked 

Planning 

estimate 

Approved 

program 

Current 

estimate 

Cost: 

Development  (RDT&E) 

Procurement 

Construction  (MILCON) 

Operations  &  maintenance  (O&M) 
Total  base  year  cost: 

Escalation 

Schedule: 

Milestone  1 

Milestone  II 

Development  contract  award 

Low  rate  initial  production  award 

Full  rate  production  award 

3  ACAT  IA  programs  are  MAIS  programs  defined  by  the  Assistant  Secretary  of  Defense  for 
Command,  Control,  Communications,  and  Intelligence  (ASD[C3I])  as  requiring  program  costs  in 
any  single  year  in  excess  of  $30  million  (FY96  constant  dollars)  or  total  life-cycle  costs  in  excess 
of  $360  million  (FY96  constant  dollars),  or  those  designated  by  the  ASD(C3I)  to  be  ACAT  IA. 

4  The  ACAT  IC  program  decision  authority  is  the  CAE.  The  “C”  refers  to  Component.  The 
ACAT  ID  program  decision  authority  is  the  USD(A&T).  The  “D”  refers  to  the  Defense  Acquisi¬ 
tion  Board,  which  advises  the  USD(A&T)  at  major  decision  points. 
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Table  3-1.  SAR  Planning  Matrix  (Continued) 


Item  tracked 

Planning 

estimate 

Approved 

program 

Current 

estimate 

Development,  test  &  evaluation 
start/finish 

First  delivery 

Operational,  test,  &  evaluation 
start/finish 

Milestone  III 

Initial  operational  capability 

Full  operational  capability 

Cost  Variance 

Cost  variance  (CV)  will  be  tracked  in  four  major  categories:  RDT&E,  procure¬ 
ment,  construction  (MELCON),  and  O&M,  and  in  total  for  both  test  and  control 
groups.  By  tracking  the  component  CVs  in  the  four  major  categories  and  in  the 
aggregate,  variations  (either  up  or  down)  that  can  be  attributed  to  incentives  may 
be  discovered.  Because  it  is  conceivable  that  increased  RDT&E  costs  may  result 
in  lower  production  costs  to  yield  a  net  program  cost  reduction,  the  primary  meas¬ 
ure  will  be  total  costs. 

CV  will  be  expressed  as  a  ratio  of  actual  costs  to  planned  costs  for  each  test  and 
control  group,  as  computed  in  Equation  3-1: 

actual  costs 

CV=— - - - .  [Eq.  3-1] 

planned  costs 

Values  of  CV  less  than  1  represent  programs  whose  actual  expenditures  are  below 
planned  levels.  Table  3-2  summarizes  the  data  necessary  to  compute  the  CVs  for 
the  enhanced  incentives  test.  All  of  the  required  data  are  recorded  in  the  SARs. 

Table  3-2.  Cost  Variance  Data  Requirements 


Cost  category 

Planned  costs 
(test  baseline) 

Actual  costs 
(current  estimate) 

Cost  variance  (CV) 

RDT&E 

Procurement 

MILCON 

O&M 

Total 
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Schedule  variance  (SV)  will  be  measured  in  a  manner  similar  to  CV.  Current  es¬ 
timates  for  milestones  that  appear  in  the  APB  will  be  captured  for  test  and  control 
groups  when  the  test  is  initiated.  For  these  groups,  a  baseline  phase  duration  (in 
months)  for  the  current  and  remaining  phases  of  the  program  will  be  calculated  as 
time  between  the  initiating  and  terminating  phase  milestones. 

SV  will  be  expressed  as  the  ratio  of  current  (actual)  to  planned  duration,  as  in 
Equation  3-2.  Current  and  remaining  phase  duration  estimates  will  be  calculated. 
Other  significant  milestones,  such  as  low  rate  initial  production  or  contract  award 
dates  that  appear  in  the  APB  or  the  SARs,  will  also  be  used  to  measure  SV. 


current  phase  duration(months ) 
planned  phase  duration(months) 


[Eq.  3-2] 


Values  of  SV  less  than  1  represent  programs  that  are  progressing  faster  than 
planned  in  achieving  schedule  milestones.  All  data  used  to  measure  SV  can  be 
found  in  the  SARs. 


Expected  Results 

In  terms  of  the  response  measures  CV  and  SV,  our  hypothesis  is  that  programs 
using  the  enhanced  incentives  system  will  have  lower  cost  and  schedule  variance 
than  those  not  using  the  enhanced  incentive  system.  This  is  illustrated  in  Figure 
3-1. 


Figure  3-1.  Notional  Example  of  Expected  Cost  and  Schedule  Variance 
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This  chart  presents  notional  results  for  10  program  offices  at  the  end  of  a  one-year 
measurement  period.  Test  sites  Ti  through  T5  are  using  the  enhanced  incentives 
system,  while  control  sites  Ci  through  C5  are  not.  The  plot  is  characterized  by  four 
quadrants.  Quadrant  I  represents  those  programs  in  which  actual  cost  and  schedule 
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are  less  than  planned.  Quadrant  II  shows  those  programs  in  which  cost  is  less  than 
planned  but  the  schedule  is  exceeded.  Quadrant  HI  depicts  those  programs  in 
which  both  cost  and  schedule  exceed  baseline  plans.  Finally,  quadrant  IV  indi¬ 
cates  those  programs  in  which  baseline  cost  estimates  are  exceeded  but  schedule 
is  less  than  planned  (i.e.,  within  schedule). 

In  this  notional  example,  the  test  sites  with  lower  CV  and  SV  are  clustered.  The 
challenge  is  to  collect  enough  sample  data,  while  controlling  for  extraneous  vari¬ 
ables,  to  provide  statistical  evidence  that  incentives  can  have  a  positive  impact  on 
our  measures  for  program  success  (CV  and  SV). 

Determining  Sample  Size 

In  order  to  determine  how  many  pilot  sites  should  participate  in  the  enhanced  in¬ 
centives  test,  four  important  elements  should  be  considered: 

♦  Historical  variation  in  the  response  measure. 

♦  The  number  of  factors  or  factor  levels  that  will  be  tested. 

♦  The  analytical  framework. 

♦  The  amount  of  risk  we  are  able  to  accept  or  afford. 

Each  of  these  elements  is  discussed  in  detail  in  the  following  sections. 

Historical  Variance  of  the  Response  Measure 

For  the  purpose  of  the  test,  we  will  look  at  the  effect  of  incentives  in  terms  of  the 
two  response  variables,  CV  and  SV.  Performance  variances  will  not  be  used 
because  research  has  shown  that  acquisition  programs  usually  meet  their 
performance  objectives.  The  question  becomes — at  what  cost  and  with  what 
regard  to  schedule? 

Program  cost  and  schedule  variances  have  been  the  subject  of  numerous  studies 
and  analyses  over  the  past  several  years.  Since  more  cost  data  were  readily 
available,  we  sized  the  sample  on  the  basis  of  available  cost  data  only,  using  the 
Defense  System  Cost  Performance  Database  (DSCPD)  to  estimate  the  expected 
variance  for  the  response  measure  CV.  Comparable  schedule  data  were  not  readily 
available. 

The  expected  variance  in  our  response  measure  is  a  key  parameter  needed  to  de¬ 
termine  how  much  data  we  must  collect  and  thus  how  many  pilot  sites  must  par¬ 
ticipate  in  the  test.  The  more  our  response  measure  varies  (even  prior  to  the 
introduction  of  incentives),  the  more  difficult  it  will  be  to  detect  statistical  differ¬ 
ences  attributable  to  incentives. 
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RAND’s  DSCPD  compiled  actual  and  planned  costs  from  ACAT I  program  SARs 
from  1969  to  1991. 5  A  program  typically  has  three  different  SAR  baselines  over 
its  life-cycle:  planning,  development,  and  production.  Planning  estimates  have  not 
always  been  included  in  the  SARs.  It  is  only  in  the  past  five  years  that  a  planning 
baseline  submittal  has  been  required  in  the  SARs.  Most  programs,  particularly 
older  ones  from  the  1960s  and  1970s,  have  only  development  estimates.  Table  3-3 
shows  the  distribution  of  CV  estimates  in  the  DSCPD. 

Table  3-3.  Distribution  of  CV 
Estimates  by  Baseline 


Baseline 

Number 

Planning  estimate 

37 

Development  estimate 

150 

Production  estimate 

88 

Total 

275 

The  cost  data  in  the  DSCPD  are  adjusted  for  changes  in  inflation  and  quantity 
changes,  because  most  analysts  feel  that  unanticipated  inflation  and  quantity 
changes  are  largely  beyond  the  program  manager’s  control.  For  each  baseline,  we 
calculated  the  CV,  standard  deviation,  and  a  95  percent  confidence  interval  in  or¬ 
der  to  determine  how  much  variation  we  might  expect  in  our  response  measure. 
The  results  are  presented  in  Table  3-4. 

Table  3-4.  Cost  Variance,  All  Programs  ( 1969-1991 ) 


Baseline 

Average 

CV 

Standard 

deviation 

95%  confidence 
interval 

Planning 

0.48 

0.81  to  1.59 

Development 

1.3 

0.42 

1.09  to  1.51 

Production 

1.04 

0.17 

1.00  to  1.08 

Total 

1.2 

0.39 

1.06  to  1.34 

These  results  indicate  that  CVs  have  averaged  1.2,  1.3,  and  1.04  respectively  for 
planning,  development,  and  production  baselines  from  1969  to  1991.  Said  another 
way,  historical  costs  have  exceeded  planned  costs  by  20  percent,  30  percent,  and  4 
percent  respectively  for  each  baseline.  The  standard  deviation  measures  how 
widely  dispersed  the  data  are  from  the  sample  mean.  The  larger  the  standard  de¬ 
viation,  the  more  dispersed  the  data  lie  about  the  mean,  and  the  wider  the  confi¬ 
dence  interval  usually  becomes.  The  more  dispersed  the  data  collected  are 


5  The  Defense  System  Cost  Performance  Database,  Cost  Growth  Analysis  Using  Selected  Ac¬ 
quisition  Reports,  J.M.  Jarvaise,  J.A.  Drezner,  D.  Norton,  RAND,  National  Defense  Research  In¬ 
stitute,  1996. 
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expected  to  be,  the  more  difficult  it  will  be  to  detect  and  attribute  changes  in  the 
response  measure  (CV)  to  the  intervention  (incentives). 

Table  3-4  also  shows  that  on  average,  total  costs  are  expected  to  exceed  planned 
costs  by  20  percent  even  after  adjusting  for  quantity  and  inflation  changes.  We  can 
state  with  95  percent  confidence  that  cost  overruns  will  be  somewhere  between  6 
percent  and  34  percent. 

Because  the  DSCPD  dates  from  1969,  these  results  include  programs  that  have 
been  long  completed  or  terminated.  We  wondered  whether  there  might  be  some 
reduction  in  the  variance  if  we  looked  only  at  currently  active  programs.  The  re¬ 
sult  of  this  analysis  indicates  that  there  has  been  some  reduction  in  CV  since 
1969.  In  total,  the  average  CV  dropped  from  1.20  to  1.17,  and  the  standard  devia¬ 
tion  dropped  for  each  baseline.  These  results  were  encouraging.  The  reduced  total 
standard  deviation  of  0.3 1  will  be  the  measure  used  to  determine  the  required 
number  of  pilot  sites  for  the  incentives  test.  Table  3-5  summarizes  the  results  of 
this  analysis. 


Table  3-5.  Cost  Variance,  Active  Programs  (DSCPD)  Only 


Baseline 

Number 

Average 

CV 

Standard 

deviation 

95% 

confidence 

interval 

Planning 

29 

1.2 

0.38 

0.77  to  1.65 

Development 

73 

1.24 

0.35 

1.04  to  1.44 

Production 

51 

1.05 

0.16 

1.01  to  1.09 

Total 

153 

1.17 

0.31 

0.99  to  1.35 

It  is  important  to  note  that  there  may  exist  even  further  reductions  in  CV  since 
1991.  The  numerous  acquisition  reform  initiatives,  including  the  Defense  Acqui¬ 
sition  Workforce  Improvement  Act,  that  have  been  enacted  since  that  time  may 
have  had  the  synergistic  effect  of  reducing  program  CV.  Because  the  DCPDS  did 
not  include  the  CV  estimates  for  programs  since  1991,  this  could  not  be  corrobo¬ 
rated,  however,  it  would  be  valuable  to  study  changes  in  CV  since  1991.  Further 
reductions  in  CV  would  increase  the  probability  that  the  effects  of  enhanced  in¬ 
centives  on  program  success  as  measured  by  CV  can  be  detected  within  the  con¬ 
text  of  this  test. 

Factor  Levels 

Another  important  element  in  determining  the  number  of  pilot  sites  necessary  to 
run  the  test  is  how  many  factor  levels  should  be  tested.  The  test  design  considers 
only  one  factor  of  interest — incentives.  At  a  minimum,  at  least  two  factor  levels 
are  needed  to  conduct  the  test:  enhanced  incentives,  and  no  incentives  (no  en¬ 
hancements  to  the  current  system).  In  general,  the  more  levels  of  the  factor  of 
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interest  (incentives)  to  be  analyzed  and  distinguished  between,  the  more  test  par¬ 
ticipants  will  be  required. 

Additional  factor  levels  of  interest  in  this  environment  are  monetary  and  non¬ 
monetary  incentives.  One  test  group  will  use  nonmonetary  incentives  only.  Factor 
levels  within  the  monetary  incentives  category  include  whether  or  not  the  test 
should  attempt  to  distinguish  between  varying  levels  of  monetary  incentives  in  an 
effort  to  determine  whether  larger  performance  incentives  lead  to  larger  perform¬ 
ance  gains  or  increase  the  probability  of  program  success. 

In  effect,  the  decision  can  be  made  within  the  analytical  framework  upon  which 
the  test  is  built.  The  decision  of  how  many  factor  levels  to  test  is  also  influenced 
by  affordability  and  risk.  These  issues  are  discussed  in  the  next  two  sections. 

Analytical  Framework 

Analysis  of  the  test  data  will  follow  a  typical  single-factor  analysis  of  variance 
(ANOVA)  model.  This  will  be  accomplished  in  two  steps: 

♦  Determine  whether  or  not  the  factor-level  means  are  the  same. 

♦  If  they  are  not  the  same,  examine  how  they  differ  and  what  the  implica¬ 
tions  of  the  differences  are. 

In  statistical  terms  we  can  describe  the  test  in  terms  of  two  hypotheses:  the  null 
hypothesis,  which  we  denote  as  Ho  and  the  alternative  hypothesis,  which  we  de¬ 
note  as  Ha.  There  are  two  sets  of  these  hypotheses,  one  relating  to  CV  and  the 
other  to  SV. 

For  each  incentive  level  that  we  will  test,  we  calculate  the  average  CV  and  SV  for 
pilot  programs  operating  at  that  incentive  level.  We  will  denote  these  averages  as 
ACV  and  ASV.  Accordingly,  ACVj  represents  the  average  CV  for  all  programs 
operating  at  incentive  level  one,  while  ASVi  represents  the  average  SV  for  all 
programs  operating  at  incentive  level  one. 

With  these  definitions,  the  following  sets  of  hypotheses  can  be  formulated: 

(1)  Cost 

Ho:  ACVi  =  ACV2  =  . . .  =  ACVj 


Ha:  Not  all  ACVj  are  equal 

Where  j  equals  the  number  of  incentive  levels  we  wish  to  test. 

The  null  hypothesis  states  that  regardless  of  incentives,  CVs  are  the  same.  The 
alternative  hypothesis  states  that  average  CV  is  influenced  by  incentives. 
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(2)  Schedule 


Ho:  ASV!  =  ASV2  =  . . .  =  ASVj 


Ha:  Not  all  ASVj  are  equal 

Where  j  equals  the  number  of  incentive  levels  we  wish  to  test. 

The  null  hypothesis  states  that  regardless  of  incentives,  SVs  are  the  same.  The  al¬ 
ternative  hypothesis  states  that  average  SV  is  influenced  by  incentives. 

It  is  important  analytically  to  test  both  sets  of  hypotheses.  With  this  formulation, 
we  may  be  able  to  determine  whether  incentives  can  affect  one  aspect  of  program 
success  (either  cost  savings  or  meeting  schedule  milestones)  more  so  than  the 
other. 

The  general  hypothesis  that  incentives  can  have  a  positive  impact  on  program  suc¬ 
cess  can  be  substantiated  only  if  we  collect  enough  statistical  evidence  to  allow  us 
to  reject  Ho  for  either  the  cost  or  schedule  formulation.  If  this  is  the  case,  step  two 
will  provide  for  pair-wise  comparison  of  die  factor  averages  to  determine  whether 
the  presence  of  enhanced  incentives  provides  lower  averages  for  either  of  the  re¬ 
sponse  measures  CV  and  SV. 

Risk  Assessment 

In  general,  the  more  data  we  are  able  to  collect,  the  better  our  chances  of  detecting 
smaller  differences  between  test  and  control  sites.  The  primary  risk  in  this  envi¬ 
ronment  is  not  having  enough  samples  from  which  to  draw  valid  statistical  con¬ 
clusions.  Several  factors  contribute  to  the  assessment  of  risk.  The  most  important 
is  how  much  we  expect  the  response  variables  to  be  affected  by  the  factor  levels. 
In  other  words,  the  larger  the  impact  incentives  have  on  cost  and  schedule  meas¬ 
ures,  the  better  our  chances  of  drawing  valid  statistical  conclusions  and  the  fewer 
samples  we  will  need  to  detect  the  differences. 

Another  factor  is  how  much  statistical  error  we  are  willing  to  accept.  Two  types  of 
errors  are  associated  with  hypothesis  testing:  Type  I  errors,  called  alpha  (a)  errors, 
and  Type  II  errors,  known  as  beta  ((3)  errors.  Type  I  errors  represent  the  probabil¬ 
ity  that  Ho  is  rejected  when  Ho  is  true.  Type  II  errors  represent  the  probability  that 
Ho  is  accepted  when  Ho  is  false.  The  following  section  on  sample  size  brings  these 
factors  together  in  a  decision  framework. 

Sample  Size 

A  sample  represents  a  program  management  office  participating  in  the  test.  Our 
sample  size  analysis  used  two  assumed  levels  of  influence  that  incentives  may 
have  on  the  response  measure  (CV),  10  percent  and  20  percent.  We  also  used 
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typical  a  and  (3  error  levels  of  .05  and  .20  respectively  to  determine  the  sample 
size  requirements.  For  analysis  of  variance  problems,  it  is  important  to  plan  the 
sample  size  so  that  needed  protection  against  Type  I  and  Type  II  errors  can  be  ob¬ 
tained.6 

Figure  3-2  summarizes  the  sample  size  requirements  for  various  test  designs;  two- 
level,  three-level,  four-level,  and  five-level.  Each  row  represents  alternative  com¬ 
binations  for  testing  the  single  factor  of  interest — incentives — at  the  correspond¬ 
ing  number  of  levels.  For  example,  at  the  two-factor  level,  the  test  may  compare 
the  affects  of  no  incentives  against  nonmonetary  incentives  or  compare  non¬ 
monetary  incentives  against  a  $6,000  incentive,  etc.  Sample  requirements  are 
listed  per  factor  level  and  total  for  each  design,  relying  on  Power  Function 
Charts.7  The  10  percent  and  20  percent  columns  represent  assumptions  made  re¬ 
garding  how  much  impact  incentives  will  have  on  CV.  If  incentives  reduce  our 
response  measure  (CV)  by  20  percent,  then  we  need  16  pilot  programs  to  test  in¬ 
centives  across  four  levels  to  be  able  to  detect  statistical  differences.  The  needed 
sample  size  increases  exponentially  if  the  expected  impact  is  only  10  percent.  Un¬ 
der  this  assumption,  72  pilot  programs  would  be  needed  in  order  to  draw  valid 
statistical  conclusions. 

The  rationale  for  choosing  the  factor  levels  of  $6,000,  $10,000,  and  $25,000  for 
monetary  incentives  is: 

♦  $6,000  represents  approximately  10  percent  of  the  average  salary  of  a 
member  of  a  program  office  staff.  Supporting  research  indicates  that  10 
percent  is  the  minimum  reward  level  that  will  influence  performance. 

♦  $10,000  represents  the  maximum  amount  an  agency  can  award  without 
higher  approval. 

♦  $25,000  represents  the  maximum  award  for  government  employees 
authorized  by  OPM  policy. 


6  “Implementation  of  ANOVA  Model,”  John  Neter  and  William  Wasserman,  Applied  Linear 
Statistical  Models,  1974,  Chapter  15,  p.  492. 

7  “Power  Function  Charts  for  Specifying  Numbers  of  Observations  in  Analyses  of  Variance  of 
Fixed  Effects,”  The  Annals  of  Mathematical  Statistics,  1958,  Volume  29,  pp.  871-877. 
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Figure  3-2.  Sample  Size  Requirements 
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Test  Recommendations 


We  recommend  that  16  pilot  programs  participate  in  the  test.  This  number  of  pro¬ 
grams  will  facilitate  the  testing  of  four  factor  (incentive)  levels  and  is  the  mini¬ 
mum  required  for  collecting  enough  data  to  detect  statistical  differences  among 
the  test  programs.  Even  with  16  pilot  sites  participating,  incentives  must  impact 
cost  variances  by  20  percent  for  the  test  to  detect  and  attribute  the  effect  to  incen¬ 
tives. 

All  pilot  programs  should  have  successfully  implemented  the  common  perform¬ 
ance  measurement  system — the  Balanced  Scorecard — to  which  incentives  will  be 
linked.  The  programs  will  be  divided  into  four  groups,  each  testing  one  of  four 
levels  of  incentives.  The  first  level  would  be  the  control  group  and  use  only  ex¬ 
isting  incentives.  The  second  level  would  have  a  fenced  awards  budget  to  test  an 
enhanced  system  of  nonmonetary  awards  linked  to  team  performance.  The  third 
and  fourth  levels  would  provide  smaller  and  larger  monetary  awards,  the  larger 
amount  testing  the  notion  that  larger  rewards  lead  to  larger  performance  gains. 

The  test  period  should  be  five  years,  to  allow  time  for  the  interventions  to  affect 
program  cost,  schedule,  and  performance  outcomes  and  to  observe  the  results. 
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Chapter  4 

Implementation 


The  test  in  program  offices  should  be  implemented  in  two  stages.  First,  a  team- 
based  performance  measurement  system  will  be  installed,  providing  the  frame¬ 
work  for  translating  mission  and  strategy  into  objectives  and  measures.  By  clearly 
defining  desired  outcomes  and  the  drivers  of  those  outcomes,  senior  managers  can 
channel  the  energies,  abilities,  and  specific  knowledge  of  people  throughout  the 
organization  toward  achieving  the  long-term  goals.  Second,  incentives  will  be 
linked  to  stretch  targets  related  to  the  strategic  objectives.  The  purpose  of  incen¬ 
tives  is  to  motivate,  drive  performance,  and  focus  the  entire  team  on  common 
mission  and  strategy-based  objectives. 

A  set  of  implementation  prerequisites  for  each  phase  has  been  developed  on  the 
basis  of  research  findings,  program  office  interviews,  and  the  enhanced  incentives 
survey.  The  prerequisites  provide  the  conditions  necessary  for  proceeding  through 
the  two  implementation  stages.  Without  the  presence  of  these  conditions,  it  is  un¬ 
likely  that  the  implementation  will  be  successful. 

Stage  One — Implementing  the  Performance 
Measurement  System 

Stage  One  Prerequisites 

For  the  balanced  scorecard  to  be  effective,  it  must  reflect  the  strategic  vision  of 
the  senior  executive  group.  Merely  slapping  performance  measures  onto  existing 
processes  may  drive  local  improvement,  but  it  is  unlikely  to  lead  to  breakthrough 
performance  for  an  entire  organization.1  If  senior  executives  are  not  leading  the 
process,  they  will  continue  to  conduct  operational  reviews  emphasizing  short-term 
targets,  bypassing  and  undermining  the  fundamental  rationale  for  developing  a 
scorecard  in  the  first  place.  It  is  through  the  involvement  and  observed  actions  of 
leadership  that  employees  sense  commitment,  urgency,  and  authenticity  with  re¬ 
spect  to  reform  initiatives. 


1  The  Balanced  Scorecard,  Translating  Strategy  into  Action,  Robert  S.  Kaplan  and 'David  P. 
Norton,  Harvard  Business  School  Press,  1996. 
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The  prerequisites  to  stage  one  are  that  leadership  at  all  levels  must  be  fully  en¬ 
gaged  and  leadership  at  the  corporate  level  must 

♦  agree  on  desired  outcomes, 

♦  clarify  and  translate  the  vision  and  strategy, 

♦  communicate  and  link  strategic  objectives  and  measures,  and 

♦  link  incentives  to  stretch  targets  related  to  the  strategic  objectives. 

Implementing  the  Balanced  Scorecard 

The  balanced  scorecard  provides  a  framework  for  communicating  organizational 
strategy,  measuring  performance  against  clearly  defined  goals  aligned  with  strat¬ 
egy,  and  linking  rewards  to  focus  behavior  and  drive  performance.  Appendix  B 
provides  a  detailed  description  of  the  balanced  scorecard.  This  section  describes 
an  implementation  plan  for  installing  the  scorecard  in  a  single  business  unit  or 
program  office. 

Implementation  is  a  four-step  process:  defining  the  organizational  structure, 
building  consensus  around  strategic  objectives,  selecting  and  designing  the  meas¬ 
ures  of  performance,  and  building  the  implementation  plan.  Each  step  is  broken 
down  into  a  set  of  subtasks.2 

Defining  the  Organizational  Structure 

The  corporate  and  business  unit  structure  for  DoD’s  weapon  (or  software)  system 
procurement  process  must  first  be  established.  The  first  two  tasks  will  ensure  that 
a  consensus  regarding  organizational  relationships  is  achieved. 

Task  1:  Select  the  Appropriate  Organizational  Unit 

The  scorecard  process  works  best  at  organizational  levels  that  have  their  own 
customers,  resources,  products,  distribution,  and  production  (e.g.,  an  acquisition 
program  office). 

Corporate-level  scorecards  ensure  that  higher  level  strategy  and  objectives  drive 
the  performance  at  the  business  unit  or  program  management  team  level.  The  di¬ 
versity  of  DoD’s  acquisition  “corporation”  increases  the  difficulty  of  constructing 
corporate-level  scorecards.  However,  the  corporate-level  scorecard  must  be  con¬ 
structed  first  to  ensure  that  the  corporate  and  business  unit  strategies  are  aligned. 


2  This  section  draws  heavily  on  research  done  by  Robert  S.  Kaplan  and  David  P.  Norton  and 
documented  in  their  book  The  Balanced  Scorecard,  Translating  Strategy  into  Action,  Harvard 
Business  School  Press,  1996,  Appendix  A,  p.  294. 
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The  corporate  scorecard  may  consist  of  some  strategy  vectors  and  a  core  set  of 
measurement  criteria. 

Task  2:  Identify  Corporate,  Division,  and  Business  Unit  Linkages 

DoD’s  acquisition  “corporation”  is  a  complex  structure  of  various  managing  enti¬ 
ties  including  USD(A&T)  staff,  CAEs,  Component  Material  and  System  Com¬ 
mands,  PEOs,  PMTs,  and  IPTs.  These  entities  must  be  identified  and  aligned  in  an 
organizational  structure  such  as  the  one  shown  in  Figure  4-1. 

Figure  4-1.  Notional  Acquisition  Organizational  Structure 


It  is  necessary  to  understand  the  relationships  between  these  entities  and  then  de¬ 
termine  at  what  levels,  and  to  what  degree,  scorecards  should  be  implemented.  For 
example,  corporate-level  scorecards  may  contain  only  the  key  strategy  vectors, 
with  a  minimal  measurement  framework.  PEOs  may  be  aligned  with  the  corporate 
level  or  may  constitute  the  first  level  to  which  detail  is  added  in  the  four  manage¬ 
ment  perspectives:  financial,  customer,  internal  processes,  and  learning  and 
growth.  Interviews  with  key  corporate,  PEO,  and  PMT  managers  will  help  define 
the  organizational  relationships. 

Leadership  at  the  corporate  and  program  office  level  should  establish  a  project 
leader  and  design  team  led  by  a  senior  staff  manager  who 

♦  owns  and  maintains  the  framework,  philosophy,  and  methodology  for  de¬ 
sign  and  development;  and 

♦  guides  the  process,  schedules  interviews  and  meetings,  synthesizes  inputs, 
and  provides  documentation,  background  reading,  and  data  to  the  project 
team. 
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Building  Consensus  Around  Strategic  Objectives 


Task  3:  Conduct  First  Round  of  Interviews 

During  the  first  round  of  interviews,  the  objectives  are  to  introduce  the  concept  of 
the  balanced  scorecard  to  senior  managers;  begin  the  process  of  having  top  man¬ 
agement  think  about  translating  strategy  and  objectives  into  tangible,  operational 
measures;  learn  about  concerns  that  individuals  may  have  about  developing  and 
implementing  the  scorecard;  and  identify  potential  conflicts  in  strategy  or 
organizational  functions. 

The  project  leader  prepares  and  supplies  background  scorecard  material  and  pro¬ 
gram-specific  documents  on  the  vision,  mission,  and  strategy  to  each  of  the  senior 
managers  who  will  be  involved  (usually  6  to  12).  Interviews  of  60  to  90  minutes 
are  conducted  to  get  inputs  on  strategic  objectives  and  tentative  proposals  on 
measures  across  the  four  balanced  scorecard  perspectives. 

Task  4:  Synthesis  Session 

After  the  interviews  are  completed,  the  design  team  meets  to  discuss  the  proposed 
strategic  objectives  and  measures.  The  outcome  of  the  synthesis  session  is  an  ini¬ 
tial  ranking  of  objectives  in  each  perspective.  These  will  be  used  as  a  basis  for  the 
first  executive  workshop. 

Task  5:  Executive  Workshop  (Round  One) 

The  senior  management  group,  with  the  assistance  of  a  facilitator  knowledgeable 
in  the  balanced  scorecard  development  process,  debates  mission  and  strategy  until 
a  consensus  is  reached.  Each  individual  then  asks,  “If  I  succeed  with  my  vision 
and  strategy,  how  will  my  performance  differ  for  customers,  internal  business 
processes,  and  our  ability  to  grow  and  improve?”  Each  objective  and  measure  is 
addressed  sequentially  for  each  perspective.  After  all  have  been  discussed,  a  vote 
is  taken  to  determine  the  top  3  or  4  candidates.  If  time  permits,  additional  meas¬ 
ures  may  be  brainstormed.  The  workshop  concludes  by  dividing  the  executive 
team  into  four  subgroups  for  the  next  task,  one  subgroup  for  each  perspective.  A 
lead  for  each  subgroup  is  chosen.  Subgroups  should  expand  to  4  to  6  persons  each 
and  include  key  functional  personnel  from  the  next  subordinate  level  of  manage¬ 
ment. 

Selecting  and  Designing  the  Measures 

Task  6:  Subgroup  Meetings 

The  challenge  in  choosing  specific  scorecard  measures  is  to  select  those  that  best 
communicate  what  the  strategy  is  intended  to  accomplish.  During  this  task,  the 
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project  lead  works  with  the  individual  subgroups  over  the  course  of  several  meet¬ 
ings  to 

♦  refine  strategic  objectives; 

♦  for  each  objective,  identify  measures  that  best  capture  and  communicate 
the  intent; 

♦  for  each  measure,  identify  sources  of  information  and  action  needed;  and 

♦  for  each  perspective,  identify  the  influences  and  linkages  between  meas¬ 
ures. 

The  facilitator  may  be  used  along  with  the  project  lead  to  begin  to  identify  the 
framework  for  the  four  perspectives.  Linkages  between  measures  (both  within  and 
across  perspectives)  that  describe  the  cause-and-effect  relationship  are  identified. 

Task  7:  Executive  Workshop  (Round  Two) 

During  this  workshop,  the  senior  management  team,  direct  subordinates,  and  a 
large  number  of  key  middle  managers  are  involved  to  achieve  buy-in.  Output  from 
the  subgroups  is  briefed  by  the  executive  leads  to  build  ownership.  Comments  on 
measures  are  taken.  The  output  of  this  session  is  a  consensus  on  stretch  goals  and 
targets  for  each  measure,  a  draft  implementation  plan,  and  vehicles  (e.g.,  bro¬ 
chures,  Web  page)  for  communicating  the  balanced  scorecard  to  employees. 

Building  the  Implementation  Plan 

Task  8:  Develop  the  Implementation  Plan 

A  new  team  of  leaders  from  the  subgroups  formalizes  the  stretch  targets.  The  im¬ 
plementation  plan  includes  how  the  measures  are  to  be  linked  to  database  and  in¬ 
formation  systems  in  order  to  communicate  the  balanced  scorecard  throughout  the 
organization. 

Task  9:  Executive  Workshop  (Round  Three) 

A  final  consensus  on  the  vision,  objectives,  and  measures  is  reached.  Stretch  tar¬ 
gets  are  validated.  An  action  plan  detailing  initiatives  for  achieving  stretch  targets 
is  constructed. 

Task  10:  Finalize  the  Implementation  Plan 

An  implementation  plan  is  finalized  for  communicating  the  scorecard  to  employ¬ 
ees,  integrating  the  balanced  scorecard  into  management  philosophy,  and  devel¬ 
oping  a  supporting  information  system.  Program  offices  should  begin  executing 
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the  implementation  plan  and  using  the  scorecard  as  soon  as  possible  with  the  best 
available  information  and  let  the  systems  catch  up  with  the  process. 

Stage  two — Linking  Incentives 

Stage  Two  Prerequisites 

Prerequisites  for  stage  two  are  specific  to  implementing  incentives  in  the  major 
defense  acquisition  program  environment.  Stage  two  should  not  be  initiated  until 
stage  one  has  been  completed.  Incentives,  particularly  monetary,  are  designed  to 
focus  and  change  behavior,  and  it  is  important  that  the  measures  capture  desired 
outcomes  and  that  process  relationships  be  understood.  Employees  and  manage¬ 
ment  should  feel  comfortable  with  and  have  confidence  in  the  strategic  objectives, 
measures,  and  targets  represented  in  the  scorecard.  In  other  words,  managers 
should  feel  reasonably  certain  that  the  performance  behavior  motivated  by  the  in¬ 
centives  supports  the  strategic  objectives  and  is  the  desired  behavior. 

The  prerequisites  to  stage  two  are: 

♦  A  team-based  performance  measurement  system  is  in  place  (from  stage 
one). 

♦  A  trial  period  to  ensure  that  strategic  measures  capture  desired  outcomes 
properly  (6  to  12  months)  is  completed  and  credibility  has  been  estab¬ 
lished. 

♦  An  approved  incentive  plan  that  includes  the  employee  participation  con¬ 
ditions  is  approved. 

♦  All  team  members — including  civilians,  military,  and  SETA  contractors — 
are  eligible  to  participate  in  the  incentive  program. 

♦  Either  cash  awards  or  savings  bonds  are  available  to  all  team  members,  to 
provide  the  stimulus  needed  to  shift  to  a  performance-based  culture. 

♦  Rewards  are  equitable. 

Introducing  the  Incentives 

The  test  of  any  reward  system  is  whether  it  produces  behaviors  that  improve  per¬ 
formance  and  rewards  the  results  those  behaviors  support.  Recent  studies  have 
focused  on  the  design  and  integration  of  team  performance  management  practices 
(goal  setting,  appraising,  developing,  and  rewarding)  in  order  to  better  support 
business  performance.  Teams  enable  employees  to  collectively  enhance  organiza¬ 
tional  performance  and  achieve  key  objectives.  However,  teams  are  not  likely  to  be 
effective  if  an  organization  fails  to  back  them  up  with  a  well  designed  performance 
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management  system.  For  teams  and  individuals  to  succeed  in  attaining  their  objec¬ 
tives,  organizations  and  employees  need  a  clear  idea  of  the  results  they  are  trying  to 
achieve.  A  credible  performance  management  system  should  communicate 
organizational  strategy,  measure  performance  against  goals  aligned  with  strategy, 
and  link  rewards  to  focus  behavior  and  drive  performance. 

Communicate  Strategy 

Traditional  performance  management  systems  commonly  suffer  from  several 
shortcomings: 

♦  Goal  setting  is  not  linked  to  strategy. 

♦  Line  employees  do  not  understand  strategic  priorities. 

♦  Few  employees  receive  meaningful  feedback. 

♦  Incentives  and  rewards  are  not  linked  to  goals  and  objectives.3 

Even  private-industry  survey  evidence,  as  shown  in  Figure  4-2,  indicates  that 
linkage  between  strategy  and  critical  performance  levers  is  generally  ineffective.4 
According  to  CFO  Magazine,  only  40  percent  of  mid-level  managers  and  3  per¬ 
cent  of  line  employees  understand  the  strategic  vision;  personal  goal  setting  is 
aligned  more  with  the  annual  budget  process  than  with  organizational  strategy; 
and  incentives  and  compensation  are  linked,  not  with  strategy,  but  with  the  annual 
budget  process. 

Figure  4-2.  Private-Industry  Views  of  the  Linkage  Between 
Strategy  and  Critical  Performance  Levers 
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3  The  Strategy  Focused  Workforce ,  a  presentation  for  the  International  Productivity  and  Qual¬ 
ity  Center  Balanced  Scorecard  Conference  by  Mario  Bognanno,  Renaissance  Solutions,  Inc.,  June 
1997. 

4  CFO  Magazine ,  Renaissance  Solutions,  Inc. 
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Communication  serves  to  signal  to  all  employees  the  critical  objectives  that  must 
be  accomplished  if  an  organization’s  strategy  is  to  succeed.  Once  all  employees 
understand  high-level  objectives  and  measures,  they  can  establish  individual  ob¬ 
jectives  that  support  the  business  unit’s  global  strategy.  At  the  conclusion  of  the 
communication  and  linkage  process,  everyone  in  the  organization  should  under¬ 
stand  the  long-term  goals,  as  well  as  the  strategy  for  achieving  them.  Individuals 
will  have  formulated  local  actions  that  will  contribute  to  achieving  business  unit 
objectives,  and  all  organizational  efforts  and  initiatives  will  be  aligned  to  the 
needed  change  processes. 

Measure  Performance 

According  to  the  American  Compensation  Association  (ACA),  performance  man¬ 
agement  and  incentive  design  in  a  team  environment  have  been  evolving  in 
American  industry  since  1935.5  Three  generations  of  development  are  identified: 

♦  First  Generation  (Gainsharing) 

>-  Measures  financial  cost 

>-  Based  on  historical  standards 

>  Indefinite  term 

>-  Focuses  on  the  “shop  floor” 

>-  Examples:  Scanlon  and  Rucker  Plans. 

The  first  generation  is  exemplified  by  the  Scanlon  Plan  (1935),  probably  the  best 
known  of  the  gainsharing  plans  and  considered  the  oldest.6  Joseph  Scanlon,  presi¬ 
dent  of  a  United  Steelworkers  Union  local,  approached  the  president  of  Empire 
Steel  and  Tinplate  Company  (Mansfield,  Ohio)  with  a  plan.  The  gainsharing  plan 
he  proposed  related  labor  costs  to  sales  value  of  production,  using  a  pre¬ 
determined  formula.  Any  gains  achieved,  through  either  increased  sales  or  re¬ 
duced  labor  costs,  would  be  shared  by  the  company  and  its  employees. 

♦  Second  Generation  (Productivity  Gainsharing) 

>•  Measures  standard  hours 

>■  Based  on  historical  standards 
>-  Indefinite  term 


5  “Perspectives  in  Compensation  and  Benefits,”  ACA  Journal,  Winter  1995,  Volume  4,  Num¬ 
ber  4,  p.  66. 

6  Gain  Sharing,  The  New  Path  to  Profits  and  Productivity,  John  G.  Belcher,  1991,  p.  60. 
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>  Focuses  on  the  shop  floor 

>-  Examples:  Improved  Productivity  Through  Sharing  (IMPROSHARE). 

The  second  generation  is  represented  by  IMPROSHARE.  Developed  by  industrial 
engineer  Mitchell  Fein  in  the  mid-1970s,  IMPROSHARE  uses  labor  standards  to 
determine  standard  hours  for  a  given  level  of  production  units.7  While  typically 
used  in  manufacturing,  IMPROSHARE  has  been  adapted  to  service  industries  as 
well.  The  standard  hours  are  compared  to  actual  hours  to  determine  whether  pro¬ 
ductivity  gains  have  been  achieved.  Gains  are  then  shared  between  the  company 
and  its  employees. 

♦  Third  Generation  (Goalsharing) 

>•  Measures  broad  business  goals 
>-  Based  on  future-oriented  goals 

>  Definite  term 

>-  Focuses  on  many  groups  of  employees 

>•  Examples:  goalsharing,  business  plans,  incentive  models. 

The  typical  example  of  third-generation  development  comes  from  Coming  Inc., 
which  has  been  implementing  goalsharing  since  1990.  Coming’s  plan  has  four 
defining  features: 

♦  It  is  based  on  performance  at  the  unit  level. 

♦  It  focuses  on  long-term  goals. 

♦  It  requires  continuous  improvement  in  performance. 

♦  It  is  simple. 

From  the  employees’  perspective,  rewards  have  been  significant,  averaging  9  per¬ 
cent  between  1990  and  1993  and  13  percent  in  1994 — for  most,  the  equivalent  of 
an  extra  month’s  pay.  From  the  company’s  perspective,  the  1994  savings-to-cost 
ratio  of  4.93  means  that  Coming  saved  nearly  five  times  what  it  paid  out,  signifi¬ 
cantly  higher  than  other  gainsharing  programs.8 


7  Ibid.,  p.  50. 

8  Strategic  Use  of  Goalsharing  at  Coming,  ACA,  Winter  1995. 
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Coming’s  path  to  success  drew  heavily  on  ACA-sponsored  research  detailing  the 
experience  of  46  organizations  using  alternative  reward  systems.  Coming’s  expe¬ 
rience  reinforces  the  major  findings  of  that  research: 

♦  Develop  a  clear  vision  of  the  organization’s  strategy  and  culture. 

♦  Track  programs  for  success. 

♦  Let  people  make  a  difference. 

♦  Have  a  sunset  period  for  measures  and  targets. 

♦  Allow  for  decentralization,  but  check  that  incentive  plans  conform  with 
corporate-wide  principles. 

♦  Separate  the  incentive  from  base  pay. 

♦  Keep  the  program  flexible. 

♦  Break  down  barriers  between  employees  and  between  units. 

♦  Simplify  the  system. 

♦  Celebrate  results. 

Goalsharing  as  described  by  this  example  is  an  applicable  incentive  strategy  for 
the  acquisition  program  offices.  The  Balanced  Scorecard  provides  a  framework  of 
strategy-based  goals  and  objectives  that  can  be  shared  by  the  team,  and  to  which 
incentives  can  be  linked.  The  system  that  propelled  Mobil  Oil  to  number  one  in 
the  industry  is  an  example  of  a  goalsharing  system.  The  following  section  pro¬ 
vides  a  notional  example  of  a  goalsharing  system  set  in  the  acquisition  environ¬ 
ment. 

Goalsharing  for  the  Acquisition  Environment— A  Notional  Example 

Let’s  suppose  that  in  order  to  create  a  strategic  vision  for  the  acquisition  commu¬ 
nity,  senior  leadership  has  agreed  on  five  corporate  strategic  goals  and  has  then 
developed  a  set  of  measurable  objectives  to  support  them: 

♦  Strategic  goal  1 :  Aggressively  implement  acquisition  reform  initiatives. 

>•  Increase  use  of  commercial  practices  and  distribution  systems. 

>-  More  competitive  sourcing  of  in-house  work. 

>-  Expand  use  of  COTS  products. 

>  Reduce  cycle  times. 
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♦  Strategic  goal  2:  Expand  partnership  with  commercial  industry. 

►  Integrate  civilian  and  military  research  and  development  to  create  ad¬ 
vanced  products  and  common  technology  bases. 

►  Use  flexible  manufacturing  to  produce  low-volume,  defense-unique 
items  on  the  same  production  lines  as  high-volume,  commercial  items. 

>-  Remove  barriers  to  partnership  (e.g.,  burdensome  government  cost  ac¬ 
counting  and  auditing  systems);  accommodate  the  need  for  long-term 
contractual  relationships. 

♦  Strategic  goal  3:  Shift  resources  from  infrastructure  and  support  to  mod¬ 
ernization  and  combat. 

►  Reduce  support  costs. 

>•  Capture  commercial  technology,  both  product  and  process. 

►  Focus  on  inherently  governmental  capabilities:  warfighting,  policy, 
management,  oversight;  for  all  other  activities,  use  competitive 
sources. 

♦  Strategic  goal  4:  Re-engineer  the  DoD  logistics  systems. 

►  Use  advanced  information  systems  and  rapid  transportation. 

>•  Achieve  visibility  of  material  assets. 

►  Reduce  order-to-receipt  time. 

♦  Strategic  goal  5:  Improve  training  and  education  of  the  acquisition  work¬ 
force. 

The  objectives  that  support  the  strategic  themes  strike  a  balance  along  the  four 
perspectives  of  the  balanced  scorecard,  as  shown  in  Figure  4-3.  For  example,  in  a 
typical  program  office,  the  customers  may  include  prime  contractors,  system  us¬ 
ers,  the  Office  of  the  Secretary  of  Defense,  and/or  Congress.  Strategic  objectives 
in  the  customer  perspective  may  include  removing  barriers  to  industry  partnership, 
achieving  system  performance  objectives,  and  achieving  schedule  milestones. 
Each  of  these  objectives  is  designed  to  create  value  for  a  specific  customer.  Other 
objectives  capture  the  strategic  themes  across  the  balance  of  the  remaining  per¬ 
spectives. 
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Figure  4-3.  Notional  Balanced  Scorecard  Example 


Perspective 

Strategic  objectives 

Strategic  measures 

Targets 

Customer 

Remove  barriers  to  partnership 
Achieve  system  performance 
objectives  (vs.  threshold) 

Achieve  schedule  milestones 

•  Average  term  of  contracts 

•  Annual  contract  audits 

•  Weight/speed/capacity  objectives 

•  Contract  Delivery  Requirements 
completed,  EMD  contract  award,  initial 
operational  test  &  evaluation  completed 

+  12  months 
- 10% 

Obj  per  APB 
- 12  months 

Internal 

business 

processes 

Capture  commercial  technology 
Integrate  R&D  facilities 

Reduce  cycle  times 

Achieve  asset  visiblity 

Competitively  outsource  in-house 
work 

•  %  product  req.  filled  by  COTS 

•  %  of  integration  plan  achieved 

•  Months  from  concept  to  initial 
operational  capability 

•  Average  order-to- receipt  time 

•  %  of  material  assets  tracked  in  system 

•  $  value  of  work  outsourced 

•  Full-time  equivalents  per  capita 

+10% 

50% 

-  24  months 

-  6  months 
00% 

+10% 

-1 

Learning  & 
growth 

Develop  strategic  skills  & 
competencies 

Utilize  advanced  information  systems 
Improve  employee  satisfaction 

•  %  of  employee  education  &  training  plan 
completed 

•  #  days  training  per  employee  per  year 

•  %  of  technology  upgrade  plan  completed 

•  Climate  survey  satisfaction  level 

•  Employee  turnover  rates 

50% 

-  20  days 

-  6  months 
90% 

+10% 

-10% 

Financial 

Reduce  support  costs 

Reduce  per-unit  costs 

Reduce  program  opeating  costs 

•  Actual/planned  O&M  costs 

•  Program  acquisition  unit  costs 

•  Average  procurement  unit  costs 

•  Operating  costs  per  employee 

.9 

- 10% 

- 10% 

- 10% 

Strategic  measures  for  each  objective  must  measure  performance  outcomes  to 
provide  focus  for  the  organization.  When  aligned  properly,  customer  objectives 
and  measures  will  drive  internal  business  processes  needed  to  support  the  custom¬ 
ers.  These  internal  processes  then  determine  the  requirement  for  employee  learn¬ 
ing  and  growth  and  for  information  systems  innovation.  Finally,  these  value 
chains  affect  the  bottom  line  in  terms  of  the  financial  measures. 

The  Incentive  Plan 

Each  program  office  will  be  required  to  draft  an  incentive  plan  for  approval.  The 
incentive  plan  should  be  drafted  during  stage  one  and  refined  during  the  trial  pe¬ 
riod  when  the  scorecards  are  in  place  and  the  strategic  measures  are  being  ob¬ 
served,  prior  initiating  stage  two.  The  incentive  plan  must  include  consideration  of 
organizational  levels;  employee  participation;  the  relative  importance  of  each  per¬ 
spective,  based  on  program  maturity;  the  relative  importance  of  each  measure 
within  each  perspective,  based  on  program  dynamics;  pay-outs;  frequency  of 
awards;  and  performance  ranges  for  each  measure. 

The  performance  range  for  each  measure  may  include  low,  average,  and  high  per¬ 
formance  targets,  or  thresholds  and  objectives.  Best-in-class  benchmarking  could 
be  incorporated,  if  applicable,  to  help  to  set  high-end  performance  ranges.  Oper¬ 
ating  initiatives  and  action  plans  should  be  drafted  to  achieve  high-end  perform¬ 
ance  ranges.  Scales  within  each  performance  range  can  determine  pay-outs  for 
each  performance  level. 

A  weighting  system  can  be  used  to  weight  performance  at  each  organizational 
level,  the  relative  importance  of  each  perspective,  and  measures  within  each  per¬ 
spective.  The  system  of  weights  and  measures  would  be  reviewed  on  an  annual 
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Implementation 


basis  to  ensure  that  the  proper  emphasis  and  strategy  are  being  communicated 
through  the  performance  measurement  system.  Action  plans  and  operating  initia¬ 
tives  should  be  reviewed  and  revised  accordingly. 

To  accommodate  personal  tastes  and  facilitate  the  incorporation  of  nonmonetary 
awards,  the  incentive  program  may  result  in  shares  or  points  being  awarded  to  the 
team  on  the  basis  of  the  weights,  performance  ranges,  and  performance  outcomes. 
The  shares  may  be  redeemed  by  individuals  from  a  menu  of  awards  that  may  in¬ 
clude  paid  time  off,  flexible  schedules,  education  and  training,  travel  opportuni¬ 
ties,  cash  awards,  savings  bonds,  gift  certificates,  hardware/software  upgrades, 
special  assignments,  etc. 

Employee  Participation 

An  important  part  of  the  incentive  plan  is  employee  participation  and  eligibility. 
This  portion  of  the  plan  must  cover  who  is  and  is  not  eligible,  how  to  define  the 
team,  transitioning  in  and  out  of  the  program  office,  and  training  on  the  plan. 

Each  PMT  will  develop  a  plan  tailored  to  represent  its  organization,  the  matrix 
support,  and  the  program’s  personnel  policies.  At  a  minimum,  the  following  sec¬ 
tions  should  be  included  in  the  plan: 

♦  Defining  the  team. 

♦  Who  is  eligible  to  participate,  including  pro-rated  participation? 

♦  How  people  become  ineligible  (e.g.,  less  than  satisfactory  performance 
during  rating  period)? 

♦  Transitioning  in  and  out  of  the  program  office. 

Summary 

The  implementation  of  the  incentives  test  should  be  accomplished  in  two  stages. 
First,  the  team-based  performance  measurement  system  should  be  installed.  Sec¬ 
ond,  incentives  can  be  linked  to  the  performance  measures.  Leadership  at  all  lev¬ 
els  must  be  fully  engaged  to  ensure  success. 

A  balanced  scorecard  enables  the  organization  to  align  its  management  processes 
and  focus  the  entire  team  on  implementing  the  strategy.  It  helps  achieve  consis¬ 
tency  of  vision  and  action  as  the  organization  changes  direction  and  introduces 
new  strategies  and  processes.  The  scorecard  provides  a  framework  for  managing 
the  implementation  of  strategy  while  allowing  the  strategy  itself  to  evolve  in  re¬ 
sponse  to  changes  in  the  political  and  technological  environments.9  Operating 

9  “Using  the  Balanced  Scorecard  As  a  Strategic  Management  System,”  Robert  S.  Kaplan  and 
David  P.  Norton,  Harvard  Business  Review,  January-February  1996. 
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initiatives  and  plans  are  made  for  achieving  stretch  goals.  An  incentive  plan  is  de¬ 
signed  to  motivate  the  workforce  and  reward  behaviors  that  support  the  strategic 
objectives. 

An  incentive  plan  should  include  the  following,  at  a  minimum: 

♦  Employee  participation  and  training. 

♦  Performance  targets  and  ranges. 

♦  Operating  initiatives  and  an  action  plan  for  goal  achievement. 

♦  A  weighting  system  for  organization  levels,  the  relative  importance  of 
each  perspective,  and  the  measures  within  each  perspective. 

♦  Pay-out  allocations  and  frequency. 


Chapter  5 

Final  Thoughts 


“That  which  gets  rewarded,  gets  repeated.”1 

Most  organizations  today  operate  in  a  turbulent  environment  with  complex  strate¬ 
gies  that,  though  valid  when  they  were  launched,  may  lose  their  validity  as  busi¬ 
ness  conditions  change.  After  having  implemented  a  variety  of  change  programs, 
each  with  its  own  champions  and  gurus,  and  each  competing  for  senior  leader¬ 
ship’s  time,  energy,  and  resources,  managers  are  finding  it  increasingly  difficult  to 
integrate  these  diverse  initiatives  to  achieve  strategic  goals — a  situation  that  leads 
to  frequent  disappointment  with  program  results.  While  restructuring  has  pro¬ 
duced  some  dramatic  one-time  economic  gains,  successful  organizations  have  re¬ 
alized  that  fundamental  strategic  choices  about  rewarding,  organizing,  and 
managing  people  affect  organizational  performance,2  and  they  are  growing  the  or¬ 
ganization’s  capability  to  continuously  fine-tune  strategies  and  activities  to  ac¬ 
commodate  ever-changing  business  conditions. 

Virtually  all  efforts  at  process  reengineering  and  technological  implementation 
will  fail  if  organizations  lose  sight  of  the  critical  part  played  by  the  people  who 
must  fill  the  redefined  roles,  within  the  reengineered  processes,  in  less  time  and 
with  fewer  resources.  Unless  work  is  organized  properly,  roles  are  clearly  defined, 
and  people  are  properly  focused  and  motivated,  reengineering  efforts  and  invest¬ 
ments  in  new  technology  are  unlikely  to  produce  desired  results  or  lead  to  com¬ 
petitive  advantage. 

In  the  old  environment,  multi-paged  job  descriptions  and  endless  lists  of  unique 
job  titles  confined  employees  to  “growing  their  jobs”  by  increasing  their  budgets 
and  the  number  of  employees  reporting  to  them.3  Individuals  with  bigger  jobs  get 
higher  pay,  regardless  of  their  skills  and  their  performance,  motivating  people  to 
develop  skills  that  help  them  move  up  the  hierarchy  rather  than  those  that  may  be 
critical  to  the  organization’s  success  because  they  enhance  key  organizational  ca¬ 
pabilities  and  core  competencies.  In  the  new  environment,  performance  and  in¬ 
centive  pay  plans  are  focusing  on  rewarding  individuals  for  increasing  their  skills 
and  competencies  so  they  can  add  more  value  to  their  organizations. 

1  Herbert  W.  Zagarow,  Ph.D.,  Chairman  of  the  Board,  Quality  Alert  Institute,  New  York. 
Formerly  Vice  President  of  the  Deming  Methods  Consultant  Group,  Dr.  Zagarow  has  over 

20  years  of  experience  in  strategic  planning,  performance  measurement,  and  statistical  process 
control. 

2  Rewarding,  Organizing  and  Managing  People  in  the  21st  Century:  Time  for  a  Strategic  Ap¬ 
proach.  Report  of  the  8th  QRMC,  June  30, 1997. 

3  From  the  Ground  Up:  Six  Principles  for  Building  the  New  Logic  Corporation,  Edward  E. 
Lawler  III,  Ph.D.,  copyright  1996,  Jossey-Bass  Inc.,  Publishers. 
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Research  done  by  Barry  Macy  of  Texas  Tech  University4  shows  that  increases  in 
productivity  of  30  to  40  percent  are  fully  possible  with  an  integrated  work  man¬ 
agement  strategy.  Macy’s  research  confirms  that  significant  performance  gains 
come  when  workers  are  engaged  and  have  opportunities  to  grow  and  use  their  full 
capabilities.  This  situation  requires  a  plan  that  wins  their  support  and  commit¬ 
ment.  Something  must  give  them  a  reason  to  embrace  change. 

That  something  is  usually  money.5  Even  draft  legislation  for  creating  PBOs  recog¬ 
nizes  the  reward  system’s  potential  to  affect  performance.  As  noted  in  Chapter  1, 
the  language  gives  PBOs  broad  discretion  to  design  leading  edge  reward  systems 
that  are  as  aggressive  as  any  in  the  private  sector.  It  builds  on  the  1993  GPRA 
goal-setting  and  performance  measurement  requirements,  authorizing  the  creation 
of  awards  programs  recognizing  organizational,  group,  and  individual  achieve¬ 
ments. 

Personnel  Demonstration  projects  at  the  Defense,  Commerce,  and  Veterans  Af¬ 
fairs  (VA)  Departments  reflect  the  understanding  that  the  federal  pay  system 
needs  to  be  modified  in  the  context  of  reform  initiatives  to  improve  performance. 
The  VA  pay  experiment  pays  incentives  (linked  to  a  balanced  scorecard)  to 
stimulate  improved  customer  service.  In  1997,  the  New  York  VA  regional  office 
reported  that  the  average  time  to  process  a  claim  had  been  reduced  from  264  days 
to  130  days  and  that  average  backlog  had  been  cut  from  25,000  to  12,000  claims. 

But  augmenting  the  current  federal  pay  system  on  the  basis  of  performance  im¬ 
provement  is  a  sensitive  and  potentially  disruptive  process  that  needs  to  be  care¬ 
fully  planned,  anticipating  the  reactions  of  all  the  stakeholders.6  Employees’ 
acceptance  of  a  new  system  depends  on  their  perception  of  management  and  its 
handling  of  the  current  program  as  well  as  their  sense  of  how  they  will  fare  under 
the  new  program. 

Leaders  frequently  talk  about  the  importance  to  their  organization  of  individuals 
and  teams  taking  ownership  of  the  processes  and  needed  improvement 
(empowerment).  At  the  same  time,  however,  their  actions  send  clear  messages 
about  the  hazards  of  risk-taking,  as  they  strip  power  from  those  who  take  risks  that 
do  not  pay  off.  The  ability  of  an  organization  to  react,  to  be  quick  and  flexible 
during  tumultuous  times,  can  hinge  on  its  ability  to  truly  empower  its  workforce. 
Plans  for  any  new  program  should  include  a  proactive  strategy  for  selling  employ¬ 
ees  on  the  need  for  change  and  the  rationale  for  the  new  program. 

As  a  result  of  acquisition  reform,  employees  are  being  encouraged  to  take  responsi¬ 
bility  for  their  individual  growth  and  advancement,  to  perform  more  diverse  tasks, 
and  to  be  more  creative  when  challenges  arise.  Innovative  companies  are  developing 


4  “Eyes  on  the  Prize,”  by  Howard  Risher,  Government  Executive,  September  1997. 

5  “Eyes  on  the  Prize,”  by  Howard  Risher,  Government  Executive,  September  1997. 

6  Ibid.,  p.  79. 
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programs  to  focus  employees  on  their  strategic  objectives  in  order  to  motivate  break¬ 
through  improvements  in  such  critical  areas  as  product,  process,  and  customer 
development.  These  programs  include  reengineered  performance  management  and 
incentive  pay  plans  that  emphasize  the  link  between  pay  and  performance. 

Implementing  enhanced  incentives,  or  pay  for  performance,  in  a  limited  number 
of  acquisition  programs  offers  a  unique  opportunity  to  adapt  and  test  an  effective 
industry  strategy  of  linking  incentive  pay  to  organizational  performance.  To  fully 
leverage  the  power  of  this  approach,  DoD  should  make  every  effort  to  obtain 
authority  to  provide  performance-based  incentive  pay  to  all  members  of  the  inte¬ 
grated  product  teams  participating  in  the  test,  including;  military,  civilians,  and 
SETA  support  contractors.  By  truly  managing  and  rewarding  the  team  as  a  collec¬ 
tive  and  entrepreneurial  entity,  breakthrough  performance  gains  and  achievements 
can  be  realized. 

The  test  described  in  this  plan  also  satisfies  FASA’s  requirement  to  provide  a  system 
of  enhanced  incentives  to  facilitate  the  achievement  of  cost,  schedule,  and  perform¬ 
ance  goals.  Using  pilot  programs  to  test  the  hypothesis  that  enhanced  incentives  will 
improve  the  likelihood  of  program  success  allows  the  opportunity  to  gain  insight  and 
to  leverage  lessons  learned.  The  test  design  provides  two  things:  evidence  for  deter¬ 
mining  the  potential  of  enhanced  incentives  to  improve  performance,  and  a  frame¬ 
work  for  implementing  pay  for  performance  efficiently  and  on  a  larger  scale. 
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Survey  Results 

The  Acquisition  Workforce  Enhanced  Incentive  Survey  was  distributed  to  nearly 
1,500  military  members,  civilian  personnel,  and  science,  engineering,  technical, 
and  administrative  (SETA)  support  contractors  in  Army,  Navy,  Air  Force,  and 
joint  program  offices,  and  to  senior  acquisition  students  at  the  Industrial  College 
of  the  Armed  Forces  and  the  Defense  Systems  Management  College.  The  respon¬ 
dents  were  31  percent  military,  55  percent  civilian,  and  14  percent  SETA  con¬ 
tractors.  Approximately  two-thirds  of  those  surveyed  were  working  in  ACAT I 
program  offices  and  one-third  were  students.1 

The  survey  had  three  main  objectives:  to  determine  what  rewards  are  valued  most, 
to  determine  the  credibility  and  effectiveness  of  current  performance  management 
practices  (communicating  goals  and  objectives,  performance  measurement,  and 
the  use  of  rewards  and  recognition),  and  to  get  stakeholder  input  to  the  test  and 
incentives  system  design.  The  results  are  organized  into  three  sections: 

♦  Rewards  and  recognition 

♦  Individual  and  team  performance 

♦  Critical  processes. 

Rewards  and  Recognition 

Reward  systems  are  communication  systems  that  should  reinforce  and  support  the 
kinds  of  behavior  that  improve  performance.  The  purpose  of  this  section  of  the 
survey  was  to  determine  the  perceived  value  of  current  reward  and  recognition 
systems  and  to  solicit  preferences  regarding  the  characteristics  of  a  new  system. 
The  survey  results  showed  the  following:2 

♦  74  percent  of  the  respondents  agreed  that  rewards  and  recognition  have  a 
large  potential  for  influencing  program  success  (73  percent  civilian, 

78  percent  military). 

♦  74  percent  favored  monetary  awards  over  nonmonetary  awards  (78  percent 
civilian,  66  percent  military). 

1  Figure  A-2-1  and  Table  A-2-1,  in  Annex  2,  Detailed  Survey  Results. 

2  Figure  A-2-2  and  Table  A-2-2,  in  Annex  2,  Detailed  Survey  Results. 
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♦  Only  19  percent  believed  that  current  appraisal  systems  are  credible  and 
should  be  used  to  determine  rewards  (21  percent  civilians,  16  percent 
military). 

♦  79  percent  agreed  that  for  awards  given  on  the  basis  of  team  performance, 
no  team  members  should  be  excluded  (73  percent  civilian,  87  percent 
military). 

♦  76  percent  agreed  that  awards  must  be  equitable  for  all  team  members,  in¬ 
cluding  military  members  and  SETA  contractors  (74  percent  civilian, 

79  percent  military). 

To  determine  what  types  of  rewards  would  provide  the  highest  motivational  value, 
we  also  asked  participants  to  what  extent  they  personally  valued  various  rewards. 
Consistently,  base  pay  increases,  cash  awards,  and  outstanding  performance  rat¬ 
ings  were  the  top  three,  while  productivity  upgrades,  administrative  support 
(considered  enablers  and  not  rewards),  and  gift  certificates  were  the  bottom  three. 
Table  A-l  presents  the  overall  results,  ranked  from  highest  to  lowest  value. 

Table  A-l.  Rewards,  Ranked  by  Mean 
Value  of  Response 


Rank 

Reward 

1 

Base  pay  increase 

2 

Cash  award 

3 

Outstanding  rating 

4 

Paid  time  off 

5 

Savings  bonds 

6 

Education  &  training 

7 

Assignment  preference 

8 

Flexible  work  hours  &  place 

9 

Influence  in  goal  setting 

10 

Assignment  of  high-status  tasks 

11 

Informal  recognition 

12 

Formal  recognition 

13 

Tuition  refunds 

14 

Unused  leave  sell-back 

15 

Productivity  upgrades 

16 

Administrative  support 

17 

Gift  certificates 

Individual  and  Team  Performance 

The  purpose  of  the  performance  section  of  our  survey  was  to  evaluate  the  effec¬ 
tiveness  of  current  performance  management  practices  and  to  determine  whether 
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existing  metrics  are  appropriate  for  measuring  or  tracking  individual  and  team 
performance.  Survey  evidence  revealed  that  current  performance  management 
practices  lack  credibility  and  fail  to  drive  performance  or  motivate.3 

Although  it  was  collectively  (91  percent)  agreed  that  individual  performance  has  a 
direct  impact  on  the  team’s  performance  and  contributes  to  program  success,  the 
survey  results  indicated  a  substantial  lack  of  confidence  in  individual  appraisal 
systems: 


♦  Only  18  percent  of  the  respondents  believed  that  current  appraisals  drive 
performance  and  motivate  employees  to  perform  well. 

♦  Only  34  percent  agreed  that  performance  appraisals  are  a  fair  and  accurate 
reflection  of  individual  performance. 

♦  Less  than  half  (43  percent)  agreed  that  performance  appraisals  take  into 
account  the  most  important  aspects  of  the  job. 

Many  private  sector  companies  have  realized  that  process-focused,  multifunc¬ 
tional  teams  dramatically  improve  the  way  they  deliver  products  and  services  to 
customers.  Our  survey  indicated  that  72  percent  of  respondents  agreed  that  team 
performance  is  more  important  than  individual  performance  in  influencing  pro¬ 
gram  success.  However,  team-based  performance  management  is  a  new  paradigm 
for  federal  program  offices.  There  is  no  policy  guidance  from  the  Office  of  Per¬ 
sonnel  Management  on  team  performance  metrics  or  appraisals.  During  inter¬ 
views,  program  team  members  indicated  that  since  their  teams  are  collectively 
held  responsible  for  success  or  failure,  they  should  be  rewarded  collectively.  The 
survey  results  showed  that: 

♦  less  than  half  of  the  respondents  (47  percent)  agreed  that  organizational 
strategy  or  team  goals  are  clearly  communicated  to  the  team, 

♦  only  33  percent  believed  that  team  performance  is  measured  against 
clearly  defined  goals, 

♦  only  29  percent  agreed  that  the  metrics  used  to  measure  team  performance 
reflect  critical  aspects  of  the  program,  and 

♦  only  3 1  percent  agreed  that  team  performance  measures  drive  performance 
and  motivate  the  team  to  perform  well.4 

Survey  evidence  suggests  that  team  performance  is  managed  even  less  effec¬ 
tively  than  individual  performance,  as  shown  in  Table  A-2. 


3  Figure  A-2-7  and  Table  A-2-3,  in  Annex  2,  Detailed  Survey  Results. 

4  Figure  A-2-10  and  Table  A-2-4,  in  Annex  2,  Detailed  Survey  Results. 
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Table  A-2.  Respondents’  Views  of  Team  vs.  Individual  Performance  Management 


Statement 

Team 

performance 

management 

(agree) 

Individual 

performance 

management 

(agree) 

Measures  reflect  critical  aspects  of  jobs 

29  percent 

43  percent 

Appraisals  and  metrics  are  fair  and  objective 

24  percent 

35  percent 

Organizational  strategy  is  clearly  communicated 

47  percent 

53  percent 

Objectives  and  goals  are  clearly  communicated 

48  percent 

46  percent 

Critical  Processes 

An  incentive  system  should  reward  performance  on  the  basis  of  measures  within  a 
team’s  span  of  control.  Those  processes  that  have  the  greatest  effect  on  program 
success  and  are  least  influenced  by  external  factors  become  the  best  candidates  for 
performance  measures  and  for  linking  rewards  to  performance. 

In  interviews  with  program  offices  prior  to  constructing  the  survey,  we  compiled  a 
list  of  processes  and  events  that  occur  at  various  stages  of  the  acquisition  life- 
cycle.  We  asked  the  survey  respondents  to  rate  the  extent  to  which  each  process 
has  an  impact  on  achieving  program  cost,  schedule,  and  technical  performance 
goals.  Figure  A-l  reflects  the  processes,  from  most  to  least  significant  impact. 

Figure  A-l.  Respondents’  Views  of  the  Impact  of  Critical  Processes  on  Program 

Success 


Question:  To  what  extent  do  these  processes  have  an  impact  on  acquisition  success  in  terms  of  cost, 
schedule,  and  performance? 

Significant  to  large  extent  Moderate  extent  Small  to  none 

Secure  Funding 
Stabilize  Resources 

Customer  Satisfaction 

Schedule,  Budget,  Performance 
Test  &  Evaluation 
Cost/Performahce  Tradeoffs 
Determine  Acquisition  Strategy 
Risk  Assessment 
Source  Selection 
Develop  Product  Definition 
PrOgress/Design  Reviews 
Tech  Interchange  Meetings 
Timely/Accurate  Reporting 
iBuild  Procurement  Plans 
:  Contract  Negotiations 

Determine  Contract  Deliverables 

Define  Award  Fee  Program  j 
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Stakeholder  Input 


In  an  attempt  to  isolate  those  processes  that  are  specific  to  the  government  pro¬ 
gram  management  team,  we  asked  which  team — government  or  prime 
contractor — has  the  greater  impact  on  the  successful  outcome  of  these  processes. 
Only  four  items  were  identified  as  being  primarily  under  the  control  of  the  gov¬ 
ernment  program  team:  secure  funding,  determine  acquisition  strategy,  source 
selection,  and  build  procurement  plans.  Other  processes  were  scored  as  influenced 
equally  by  both  government  and  prime  contractors. 

External  Factors 

It  is  generally  agreed  that  some  external  factors — e.g.,  budget/funding  instability 
and  schedule  or  quantity  changes — can  have  a  great  effect  on  whether  acquisition 
programs  achieve  their  cost,  schedule,  and  performance  goals.  In  order  to  deter¬ 
mine  whether  or  not  enhanced  incentives  can  contribute  to  program  success,  we 
must  be  able  to  discount  the  effects  of  such  factors,  either  mathematically  or 
through  an  arbiter  process  to  control  for  them  in  a  test  environment. 

We  tried  to  ascertain  how  much  control  the  program  office  has  over  various  ex¬ 
ternal  factors  that  may  significantly  affect  program  success.  Figure  A-2  summa¬ 
rizes  the  results. 

Figure  A-2.  Respondents’  Views  of  Program  Office’s  Control  over  External 

Influences 


Requirement  stablity  &" 
quantity  changes 

Schedule  changes 

Funding  stablity 

Accuracy  of  cost  ■ 
estimates 

Adequate  skills  & 
personnel  turnover . 

Plan  specificity 
Contractor  buy-in 
External  oversight 
Contractor  performance 


11  Substantial] 
■  Moderate 
None 


According  to  the  survey  responses,  the  program  office  has  the  least  control  over 
funding  stability,  external  oversight,  and  contractor  buy-in.  The  large  degree  of 
perceived  moderate/substantial  control  over  requirement  stability/quantity 
changes  and  schedule  changes  contradicted  interview  findings.  The  survey  re¬ 
sponses  may  reflect  the  program  office’s  ability  to  do  some  cost,  schedule,  and 
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performance  tradeoffs  under  the  cost  as  an  independent  variable  (CAIV)  philoso¬ 
phy. 

Significantly,  68  percent  of  the  respondents  believed  that  external  factors  can  pre¬ 
vent  teams  from  accomplishing  goals,  and  81  percent  felt  that  external  factors 
have  a  large  influence  on  achieving  cost,  schedule,  and  performance  goals. 

Summary 

The  survey  provided  empirical  evidence  to  support  interview  findings  that  re¬ 
wards  have  a  large  potential  to  influence  the  degree  to  which  programs  are  suc¬ 
cessful,  and  that  an  incentives  program  linked  to  a  credible  performance 
measurement  system  can  provide  an  effective  tool  for  bringing  about  positive 
change  in  the  acquisition  program  environment.  The  survey  also  indicated  that 
both  civilians  and  military  place  a  high  value  on  monetary  awards  and  that  they 
believe  that  current  performance  management  systems  lack  credibility,  that  team 
performance  is  managed  less  effectively  than  individual  performance,  and  that 
external  influences  have  a  large  impact  on  program  success. 

The  evidence  collected  to  date  through  program  office  interviews,  the  enhanced 
incentives  survey,  and  research  on  industry  practices  supports  the  recommenda¬ 
tion  to  proceed  with  a  test.  Before  monetary  or  other  extrinsic  awards  are  made, 
however,  it  is  important  to  ensure  that 

♦  a  credible  team-based  performance  measurement  system  to  link  incentives 
to  performance  results  has  been  established; 

♦  all  team  members  are  eligible  to  participate,  including  civilians,  military 
members,  and  SETA  contractors; 

♦  either  cash  awards  or  savings  bonds  are  available  to  all  team  members;  and 

♦  rewards  are  equitable. 

Annex  1  is  a  copy  of  the  survey  instrument,  while  Annex  2  tabulates  the  survey 
results  in  detail. 
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Annex  1 

Acquisition  Workforce  Enhanced  Incentives  Survey 


A-l-l 


Acquisition  Workforce  Enhanced  Incentives  Survey 


RCS:  DD-A&T(OT)2029 
EXPDT:  21  Aug  98 


Acquisition  Workforce  Enhanced  Incentives  Survey 


The  Under  Secretary  of  Defense  for  Acquisition  and  Technology  is  formulating  a  plan  to  test  an 
enhanced  system  of  incentives  for  the  acquisition  workforce.  This  survey  is  being  conducted  to 
involve  key  stakeholders  in  the  design  of  the  new  reward  system. 

The  results  of  this  survey  will  be  used  to  help  construct  the  test  to  determine  if  workforce 
incentives  can  contribute  to  acquisition  process  success  with  regard  to  cost,  schedule,  and 
performance  goals. 

Please  base  your  answers  on  your  current  (or  most  recent)  acquisition  program  management  office 
experience.  If  you  wish  to  comment  on  any  aspect  of  this  program  or  survey,  or  qualify  your 
answers  to  any  question,  please  use  the  comments  section  at  the  end  of  this  survey. 


Thank  you  for  your  help. 


PRIVACY  NOTICE 


Authority:  10  U.S.C.  136 

Principal  Purpose: 

Information  collected  in  this  survey  will 
assist  in  formulating  test  concepts  and  features 
to  determine  if  workforce  incentives  can 
contribute  to  acquisition  process  success  with 
regard  to  cost,  schedule,  and  performance  goals. 


Disclosure: 

Voluntary.  Failure  to  respond  will  not  result  in  any 
penalty  to  the  respondent.  Maximum  participation 
is  encouraged  however,  so  that  the  data  will  be 
complete  and  representative.  Your  survey  will  be 
used  only  by  persons  engaged  in,  and  for  the  purposes 
of,  the  survey.  Only  group  statistics  will  be  reported. 
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Rewards  and  Recognition 

The  following  section  addresses  current  reward  and  recognition  systems  and  asks  your  preference 
about  the  characteristics  of  a  new  system.  Please  base  your  answers  on  your  current  (or  most 
recent)  acquisition  program  management  office  experience. 

Q-l  To  what  extent  do  you  agree  or  disagree  with  the  following  statements  concerning  rewards 
and  recognition? 


Circle  the  number  of  your  answer  for  each  statement. _ 

Strongly  Neither  Agree  Strongly 

Statement  Disagree  Disagree  or  Disagree  Agree  Agree 


Current  System: 


Rewards  and  recognition  1  2  3  4  5 

have  a  large  influence  on  behavior 

The  current  performance  appraisal  1  2  3  4  5 

system  is  credible  and  appropriate 
to  determine  reward  distributions 

In  my  view,  award  fee  payments  to  the  1  2  3  4  5 

prime  contractor  are  disruptive  in  the 
IPT  environment 

New  System: 

Rewards  and  recognition  have  a  large  1  2  3  4  5 

potential  to  influence  the  success  of  the 
acquisition  process 

I  would  prefer  monetary  awards  1  2  3  4  5 

over  non-monetary  awards 

If  monetary  awards  are  given  based  on  1  2  3  4  5 

team  performance,  all  team  members 
(high  and  low  performers)  should  receive 
the  same  amount 

If  awards  are  given  based  on  team  1  2  3  4  5 

performance,  an  inequity  results  if  military 
or  contractor  (SETA)  team  members 
are  excluded  from  cash  awards 

If  awards  are  given  based  on  team  1  2  3  4  5 

performance,  all  team  members 
should  be  rewarded  equitably 
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Acquisition  Workforce  Enhanced  Incentives  Survey 


Q-2  Of  what  value  is  each  of  the  following  rewards  to  you  personally? 

Circle  the  number  of  your  answer  for  each  example. 

Moderate  Large  Significant 

Reward  No  Value  Little  Value  Value  Value  Value 


Informal  (non-ceremonial)  recognition  1  2 

by  peers,  supervisors,  DoD  leaders 

Formal  (ceremonial)  recognition  by  peers  1  2 

supervisors,  DoD  leaders 

Outstanding  performance  rating  1  2 

Increase  in  base  pay  1  2 

Paid  time  off  1  2 

Education  and  training  opportunities  1  2 

Cash  award  (>$  1 000)  1  2 

U.S.  Savings  Bonds  (>$1000)  1  2 

Productivity  upgrades  (facility,  1  2 

computers,  software,  etc.) 

Assignment  of  administrative  support  1  2 

Tuition  refunds  for  college  courses  1  2 

Follow-on  assignment  preference  1  2 

Assignment  of  high-status  tasks  1  2 

within  current  office 


3  4  5 

3  4  5 

3  4  5 
3  4  5 
3  4  5 
3  4  5 
3  4  5 
3  4  5 
3  4  5 

3  4  5 
3  4  5 
3  4  5 
3  4  5 


More  flexibility  concerning  work 
hours  and  place  (e.g.  from  home) 

Gift  certificates  for  sporting  events, 
dinner,  theater 


More  influence  in  goal  setting 
and  decision  making 


Unused  leave/vacation  time  sell-back 
Other  (Specify): 


2 

2 


3 

3 


4 

4 


Logistics  Management  Institute,  2000  Corporate  Ridge,  McLean  VA  22102-7805,  FAX  (703)  917-7180 

Phil  Lussier  (plussier@lmi.org),  Peggy  Miller  (pmiller@lmi.org)  (703)  917-7536/7406  3 


A-l-5 


Q-3  Please  rank  order  the  three  awards  that  you  would  personally  value  most  from  the  list 
below  (specify  ‘Other’  if  your  choice  does  not  appear). 

Circle  #/  for  your  first  choice ,  #2  for  your  second  choice ,  and  #3  for  your  third  choice  ( Circle  only 
three). 


Award 


Rank 


Informal  (non-ceremonial)  recognition  1  2  3 

by  peers,  supervisors,  DoD  leaders 

Formal  (ceremonial)  recognition  by  peers  1  2  3 

supervisors,  DoD  leaders 

Outstanding  performance  rating  1  2  3 

Increase  in  base  pay  12  3 

Paid  time  off  12  3 


Education  and  training  opportunities 

Cash  award  (>$1000) 

U.S.  Savings  Bonds  (>$1000) 

Productivity  upgrades  (facility, 
computers,  software,  etc.) 

Assignment  of  administrative  support 

Tuition  refunds  for  college  courses 

Follow-on  assignment  preference 

Assignment  of  high-status  tasks 
within  current  office 

More  flexibility  concerning  work 
hours  and  place  (e.g.  from  home) 

Gift  certificates  for  sporting  events, 
dinner,  theater 


1  2  3 
1  2  3 
1  2  3 
1  2  3 

1  2  3 
1  2  3 
1  2  3 
1  2  3 

1  2  3 

1  2  3 


More  influence  in  goal  setting  1  2  3 

and  decision  making 

Unused  leave/vacation  time  sell-back  1  2  3 


Other  (Specify):  1  2  3 
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Acquisition  Workforce  Enhanced  Incentives  Survey 


Q-4  In  your  opinion,  is  the  reward  more  appropriate  for  Teams  or  Individuals?  Circle  ‘Both’  if 
you  feel  that  the  reward  is  appropriate  for  Teams  and  Individuals. 

Circle  the  number  of  your  answer  for  each  example  (Please  circle  only  one  answer  for  each  reward). 

Reward  Teams  Individuals  Both 


Informal  (non-ceremonial)  recognition  12  3 

by  peers,  supervisors,  DoD  leaders 

Formal  (ceremonial)  recognition  by  peers  1  2  3 

supervisors,  DoD  leaders 

Paid  time  off  12  3 


Education  and  training  opportunities  1  2  3 

Cash  award  (>$1000)  1  2  3 

U.S.  Savings  Bonds  (>$1000)  1  2  3 

Productivity  upgrades  (facility,  1  2  3 

computers,  software,  etc.) 


Assignment  of  administrative  support  1  2  3 

Tuition  refunds  for  college  courses  1  2  3 

Assignment  of  high-status  tasks  1  2  3 

within  current  office 


More  flexibility  concerning  work  1  2  3 

hours  and  place  (e.g.  from  home) 

Gift  certificates  for  sporting  events,  12  3 

dinner,  theater 

More  influence  in  goal  setting  1  2  3 

and  decision  making 

Other  (Specify):  1  2  3 
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Individual  and  Team  Performance 

Q-5  To  what  extent  do  you  agree  or  disagree  with  the  following  statements  concerning 

performance  management  practices?  Please  base  your  answers  on  your  current  (or  most 
recent)  experience  in  an  acquisition  program  management  office. 

Individual  Performance 


Circle  the  number  of  your  answer  for  each  statement. 


Statement 


Strongly  Neither  Agree  Strongly 

Disagree  Disagree  or  Disagree  Agree  Agree 


Organizational  strategy  and  1 

visions  are  clearly  communicated 

Individual  performance  objectives  1 

are  clearly  communicated 

Performance  appraisals  are  a  fair  1 

and  accurate  reflection  of 
individual  performance 

Performance  appraisals  take  into  1 

account  the  most  important 
aspects  of  the  job 

The  current  appraisal  systems  drive  1 

performance  and  motivate 
employees  to  perform  well 

Individual  performance  directly  1 

impacts  the  team 


Individual  performance  directly  1  2  3  4  5 

contributes  to  program  success 


Individual  performance  objectives  1  2  3  4  5 

are  aligned  with  team  goals  and 

objectives 
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Acquisition  Workforce  Enhanced  Incentives  Survey 


Q-5  (continued)  To  what  extent  do  you  agree  or  disagree  with  the  following  statements 
concerning  performance  management  practices? 


Team  Performance 


Circle  the  number  of  your  answer  for  each  statement . 


Statement 


Strongly  Neither  Agree  Strongly 

Disagree  Disagree  or  Disagree  Agree  Agree 


Team  performance  is  more  1  2  3  4  5 

important  than  individual  performance 
in  terms  of  influencing  program  success 

Organizational  strategy  and  1  2  3  4  5 

visions  are  clearly  communicated 
to  the  team 

Team  goals  and  objectives  are  1  2  3  4  5 

linked  to  organizational  strategy 

Team  goals  and  objectives  are  1  2  3  4  5 

clearly  defined  and  communicated 

Team  performance  is  measured  1  2  3  4  5 

against  clearly  defined  goals 
and  objectives 

Metrics  used  to  measure  team  1  2  3  4  5 

performance  are  fair  and  objective 

Metrics  used  to  measure  team  1  2  3  4  5 

performance  reflect  the  critical 
aspects  of  the  program 

Team  performance  measures  1  2  3  4  5 

drive  performance  and  motivate 
the  team  to  perform  well 

Program  Success  Measures 

The  Acquisition  Program  Baseline  1  2  3  4  5 

(APB)  is  a  good  basis  for  measuring 
acquisition  process  success 

Deviations  from  the  APB  are  largely  1  2  3  4  5 

due  to  factors  beyond  the  control  of 
the  program  management  team 
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Critical  Processes 


Q-6  To  what  extent  does  each  process  impact  acquisition  process  success  in  terms  of  cost, 
schedule,  and  technical  performance  goals? 


Circle  the  number  of  your  answer  for  each  process . 


Process 

Not  at  all 

Small 

Extent 

Moderate 

Extent 

Large 

Extent 

Significant 

Extent 

Develop  product  definition  statements 

1 

2 

3 

4 

5 

Generate  schedule,  budget,  and 
performance  specifications 

1 

2 

3 

4 

5 

Build  procurement  plans 

1 

2 

3 

4 

5 

Cost/performance  tradeoffs 

1 

2 

3 

4 

5 

Determine  acquisition  strategy 

1 

2 

3 

4 

5 

Progress/design  reviews 

1 

2 

3 

4 

5 

Risk  assessment 

1 

2 

3 

4 

5 

Determine  contract  deliverables 

1 

2 

3 

4 

5 

Secure  funding 

1 

2 

3 

4 

5 

Stabilize  resources 

1 

2 

3 

4 

5 

Source  selection 

1 

2 

3 

4 

5 

Contract  negotiations 

1 

2 

3 

4 

5 

Define  and  implement  contractor 
award  fee  program 

1 

2 

3 

4 

5 

Test  and  evaluation 

1 

2 

3 

4 

5 

Technical  interchange  meetings 

1 

2 

3 

4 

5 

Timely  and  accurate  reporting 

1 

2 

3 

4 

5 

Customer  satisfaction 

1 

2 

3 

4 

5 

Other  (Specify): 

1 

2 

3 

4 

5 
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Acquisition  Workforce  Enhanced  Incentives  Survey 


Q-7  Which  program  team,  government  or  prime  contractor,  has  a  greater  impact  on  the 

successful  outcome  of  these  processes?  Indicate  ‘Both’  if  you  feel  that  both  teams  equally 
affect  the  successful  outcome  of  the  process. 

Circle  the  number  of  your  answer  for  each  process. 

Process  Government  Prime  Contractor  Both 


Develop  product  definition  statements  1  2 

Generate  schedule,  budget,  and  1  2 

performance  specifications 

Build  procurement  plans  1  2 

Cost/performance  tradeoffs  1  2 

Determine  acquisition  strategy  1  2 

Progress/design  reviews  1  2 

Risk  assessment  1  2 

Secure  funding  1  2 

Stabilize  resources  1  2 

Source  selection  1  2 

Contract  negotiations  1  2 

Define  and  implement  contractor  1  2 

award  fee  program 

Test  and  evaluation  1  2 

Technical  interchange  meetings  1  2 

Timely  and  accurate  reporting  1  2 

Customer  satisfaction  1  2 

Other  (Specify):  1  2 


3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 
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An  incentive  system  should  reward  performance  based  on  factors  that  are  within  program  office 
control  and  not  be  based  on  factors  that  are  external  to  the  program  office.  Below  is  a  potential 
list  of  factors  that  can  influence  the  success  of  the  acquisition  process. 

Q-8  How  much  control  does  the  Program  Management  Office  have  over  these  factors?  Please 
include  additional  factors  you  consider  important.  Use  the  comment  section  to  elaborate. 

Circle  the  number  of  your  answer  for  each  example. 


Factors 


None  Little  Moderate  Substantial  Full 

at  All  Control  Control  Control  Control 


Requirement  stability/quantity  changes  1  2 

Program  schedule  changes  1  2 

Funding  stability  1  2 

Accuracy  of  cost  estimates  1  2 

Adequate  skills/personnel  turnover  1  2 

Plan  specificity  1  2 

Contractor  bidding  strategy  1  2 

(e.g.  low  bid  to  buy-in) 

External  oversight  (e.g.  audits)  1  2 

Contractor  performance  1  2 

Other  (Specify):  _  1  2 

Other  (Specify):  _  1  2 


3  4  5 
3  4  5 
3  4  5 
3  4  5 
3  4  5 
3  4  5 
3  4  5 

3  4  5 
3  4  5 
3  4  5 
3  4  5 


Q-9  To  what  extent  do  the  above  factors  influence  cost,  schedule,  and  performance  goals? 

[]  not  at  all 
[]  small  extent 
[]  moderate  extent 
[]  large  extent 
[]  significant  extent 


Q-10 


To  what  extent  do 
team  objectives? 


the  above  factors  prevent  the  team  from  successfully  accomplishing 
[]  not  at  all 
[]  small  extent 
[]  moderate  extent 
[]  large  extent 
[]  significant  extent 
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Acquisition  Workforce  Enhanced  Incentives  Survey 


Background 

Q-ll  Are  you: 

[]  Military 

[]  Government  Civilian 
[]  Support  Contractor  (SETA) 

Q-12  How  many  years  experience  do  you  have  in  a  Program  Office? 

[]  none 
[]  less  than  4 
[]  4  to  less  than  8 
[]  8  to  less  than  12 
[]  12  or  more 

Q-13  What  is  your  Service  Department?  (contractors  may  skip  questions  14  through  17) 
[]  Army 
[]  Navy 

[]  Marine  Corps 
[]  Air  Force 
[]  Defense  Agency 

Q-14  What  is  your  current  grade? 

[]  GS  9-12 
[]  GS  13-14 
[]  GS  15 
[]  03  or  below 
[]  04-05 
[]  06  and  above 

Q-15  How  many  years  experience  do  you  have  in  Acquisition? 

[]  none 
[]  less  than  4 
[]  4  to  less  than  8 
[]  8  to  less  than  12 
[]  12  or  more 

Q-16  How  long  ago  was  your  last  program  office  assignment? 

[]  have  not  had  one 

[]  within  the  last  3  years  (or  current) 

[]  more  than  4  but  less  than  8  years  ago 
[]  more  than  8  years  ago 

Q-17  Are  you  a  member  of  the  Acquisition  Corps? 

[]  Yes 
[]  No 


Logistics  Management  Institute,  2000  Corporate  Ridge,  McLean  VA  22102-7805,  FAX  (703)  917-7180 
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Comments 


Please  use  this  space  (and  additional  sheets  as  necessary)  to  elaborate  on  your  answers  to  any  of 
the  questions  or  to  recommend  specific  strategies  that  you  think  are  important  in  the  reward  and 
recognition  of  acquisition  professionals. 


Your  contribution  to  this  survey  is  greatly  appreciated.  If  you  would  like  a  summary  of  the 
results,  please  contact  us  via  any  of  the  addresses  below  and  we  will  see  that  you  receive  the 
summary. 

Logistics  Management  Institute,  2000  Corporate  Ridge,  McLean  VA  22102-7805,  FAX  (703)  917-7180 

Phil  Lussier  (plussier@lmi.org),  Peggy  Miller  (pmiller@lmi.org)  (703)  917-7536/7406  1 2 
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Annex  2 

Detailed  Survey  Results 


This  annex  to  Appendix  A  provides  detailed  data  regarding  survey  participant 
demographics,  response  distributions  for  each  question,  statistical  differences  by 
military  service,  military  versus  civilian  personnel,  and  program  management 
team  (PMT)  members  versus  acquisition  students. 

Demographics 

Figure  A-2-1.  Respondent  Demographics  (by  program  or  school) 


^  SETA 
□  Civilian 
M  Military 


Table  A-2-1.  Respondents  by  Personnel  Type 


Military 

Civilian 

SETA 

Total 

PMT  Members 

116 

320 

115 

551 

(21  percent  of 
PMT) 

(58  percent  of 
PMT) 

(21  percent  of 
PMT) 

(67  percent  of 
Total) 

Students 

136 

133 

1 

270 

(50  percent  of 
Students) 

(49  percent  of 
Students) 

(0.4  percent  of 
Students 

(33  percent  of 
Total) 

Total 

252 

453 

116 

821 

(31  percent  of 
Total) 

(55  percent  of 
Total) 

(14  percent  of 
Total) 
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Rewards  and  Recognition 


Figure  A-2-2.  Respondents’  Views  of  Rewards  and  Recognition  (graphic) 


Note:  See  Table  A-2-2. 


Table  A-2-2.  Respondents’  Views  of  Rewards  and  Recognition  (tabular) 


Question 

Percent 

agree 

Percent 

disagree 

Percent 

neither 

Qla.  Rewards  and  recognition  have  a  large  influence  on 
behavior 

67 

19 

14 

Qlb.  The  current  appraisal  system  is  credible  and  appro¬ 
priate  to  determine  reward  distributions 

19 

55 

26 

Qlc.  In  my  view,  award  fee  payments  to  the  prime  con¬ 
tractor  are  disruptive  in  the  Integrated  Product  Team 
environment 

22 

43 

35 

Qld.  Rewards  and  recognition  have  a  large  potential  to  in¬ 
fluence  success 

74 

10 

15 

Qle.  1  would  prefer  monetary  over  non-monetary  awards 

74 

8 

18 

Qlf.  If  monetary  awards  are  based  on  team  performance, 
all  team  members  (high  and  low  performers)  should  receive 
the  same  amount 

48 

39 

12 

Q1  g.  If  awards  are  based  on  team  performance,  an  inequity 
results  if  military  of  SETA  contract  team  members  are  ex¬ 
cluded  from  cash  awards 

79 

10 

11 

Q1  h.  If  awards  are  based  on  team  performance,  all  team 
members  should  be  rewarded  equitably 

76 

14 

9 

A-2-2 


Detailed  Survey  Results 


Figure  A-2-3.  Differences  by  Service 
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Table  A-2-3.  Significance  Tests:  Rewards  and  Recognition 


Question 

Mil  v.  Civ 

PMT  v.  Students 

Rewards  &  recognition  have  large 
influence 

No  difference 

No  difference 

Appraisal  credible 

Mil  less  likely  to 
disagree  (54%  v.  62%) 

PMT  less  likely  to 
disagree  (50%  v.  67%) 

Award  fee  disruptive 

Civ  less  likely  to 
disagree  (40%  v.  50%) 

PMT  more  likely  to 
agree  (25%  v.  17%) 

Rewards  &  recognition  have  large 
potential 

No  difference 

No  difference 

Favor  monetary  over  nonmonetary 

Civ  more  likely  to  agree 
(78%  v.  66%) 

No  difference 

High/low  performers  rewarded  the 
same 

No  difference 

PMT  more  likely  to 
agree  (51  %v.  44%) 

Rewards  should  be  equitable 

No  difference 

PMT  less  likely  to 
disagree  (12%  v.  19%) 

Inequitable  if  exclusions 

Mil  more  likely  to  agree 
(87%  v.  73%) 

PMT  less  likely  to 
agree  (76%  v.  86%) 
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Figure  A-2-4,  Rewards  Ranked  and  Clustered  by  Decreasing  Value — Civilian 
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Figure  A-2-5.  Rewards  Ranked  and  Clustered  by  Decreasing  Value — Military 
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Figure  A-2-6.  Respondents’  Views  of  Individual  Performance  Management 
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Note:  See  Table  A-2-4. 


Table  A-2-4.  Respondents’  Views  of  Individual  Performance  Management 


Question 

Percent 

agree 

Percent 

disagree 

percent 

neither 

Q5a.  Organizational  strategy  and  visions  are  clearly  com¬ 
municated 

53 

27 

19 

Q5b.  Individual  performance  objectives  are  clearly  commu¬ 
nicated 

46 

30 

24 

Q5c.  Performance  appraisals  are  a  fair  and  accurate  re¬ 
flection  of  individual  performance 

35 

40 

25 

Q5d.  Performance  appraisals  take  into  account  the  most 
important  aspects  of  the  job 

43 

30 

27 

Q5e.  The  current  appraisal  systems  drive  performance  and 
motivate  employees  to  perform  well 

18 

56 

26 

Q5f.  Individual  performance  directly  impacts  the  team 

91 

3 

5 

Q5g.  Individual  performance  directly  contributes  to  program 
success 

90 

3 

7 

Q5h.  Individual  performance  objectives  are  aligned  with 
team  goals  and  objectives 

42 

22 

36 
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Table  A-2-5.  Significance  Tests:  Individual  Performance  Management 


Question 

Mil  v.  Civ 

PMT  v.  Students 

Org  strategy  clearly  communicated 

No  difference 

PMT  less  likely  to 
disagree  (24%  v.  34%) 

Indiv  performance  object  clearly 
comm 

No  difference 

PMT  less  likely  to 
disagree  (26%  v.  38%) 

Appraisals  fair  &  accurate 

Civ  more  likely  to 
disagree  (48%  v.  32%) 

PMT  less  likely  to 
disagree  (37%  v.  47%) 

Account  for  most  important  aspect 
of  job 

Civ  more  likely  to 
disagree  (35%  v.  24%) 

PMT  less  likely  to 
disagree  (27%  v.  36%) 

Current  system  drives  perform¬ 
ance  &  motivate 

Civ  more  likely  to 
disagree  (62%  v.  48%) 

PMT  more  likely  to 
agree  (21  %v.  17%) 

Individual  performance  impacts  the 
team 

No  difference 

No  difference 

Individual  performance  contributes 
to  program  success 

No  difference 

No  difference 

individual  perf  objectives  align  with 
team 

No  difference 

PMT  less  likely  to 
disagree  (14%  v.  28%) 

Figure  A-2-7.  Respondents’  Views  of  Team  Performance  Management 


All 
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Note:  See  Table  A-2-6. 
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Table  A-2-6.  Respondents’  Views  of  Team  Performance  Management 


Question 

Percent 

agree 

Percent 

disagree 

Percent 

neither 

Q5i.  Team  performance  is  more  important  than  individual 
performance  in  influencing  program  success 

72 

12 

16 

Q5j.  Organizational  strategy  and  visions  are  clearly  com¬ 
municated  to  the  team 

47 

27 

27 

Q5k.  Team  goals  and  objectives  are  linked  to  organiza¬ 
tional  strategy 

54 

16 

30 

Q5I.  Team  goals  and  objectives  are  clearly  defined  and 
communicated 

48 

22 

29 

Q5m.  Team  performance  is  measured  against  clearly  de¬ 
fined  goals  and  objectives 

33 

33 

34 

Q5n.  Metrics  used  to  measure  team  performance  are  fair 
and  objective 

24 

26 

50 

Q5o.  Metrics  used  to  measure  team  performance  reflect 
the  critical  aspects  of  the  program 

29 

23 

47 

Q5p.  Team  performance  measures  drive  performance  and 
motivate  the  team  to  perform  well 

31 

28 

40 

Table  A-2-7.  Significance  Tests :  Team  Performance 


Question 

Mil  v.  Civ 

PMT  v.  Students 

Team  performance  more 
important 

Mil  more  likely  to  agree 
(83%  v.  66%) 

No  difference 

Organizational  strategy  communi¬ 
cated  to  team 

No  difference 

PMT  less  likely  to 
disagree  (24%  v.  32%) 

Team  goals  linked  to  strategy 

Mil  more  likely  to  agree 
(62%  v.  50%) 

PMT  less  likely  to 
disagree  (14%  v.  22%) 

Team  goals  communicated 

No  difference 

No  difference 

Team  performance  measured 
against  goals 

No  difference 

PMT  less  likely  to 
disagree  (29%  v.  40%) 

Metrics  fair  and  objective 

No  difference 

PMT  less  likely  to 
disagree  (22%  v.  34%) 

Metrics  reflect  critical  aspects  of 
program 

No  difference 

PMT  less  likely  to 
disagree  (19%  v.  32%) 

Team  performance  measures 
drive  performance 

No  difference 

PMT  less  likely  to 
disagree  (27%  v.  37%) 

APB  is  good  basis  to  measure 
success 

Mil  more  likely  to  agree 
(43%  v.  34%) 

Students  more  likely  to 
agree  (45%  v.  32%) 

APB  deviations  are  beyond  con¬ 
trol 

No  difference 

Students  more  likely  to 
disagree  (21%  v.  12%) 
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Appendix  B 

The  Balanced  Scorecard 


Overview 

As  managers  and  academics  have  tried  to  remedy  the  inadequacies  of  current  per¬ 
formance  management  systems,  some  have  focused  on  making  financial  measures 
more  relevant.1  Others  have  said,  “Forget  the  financial  measures.  Improve  opera¬ 
tional  measures  like  cycle  time  and  defect  rates;  the  financial  results  will  follow.” 
But  managers  should  not  have  to  choose  between  financial  and  operational  meas¬ 
ures.  Financial  measures  tell  the  results  of  actions  already  taken,  while  operational 
measures  of  customer  satisfaction,  internal  processes,  and  the  organization’s  in¬ 
novation  and  improvement  activities  are  the  drivers  of  future  financial  perform¬ 
ance.  Senior  leadership  cannot  rely  on  one  set  of  measures  to  the  exclusion  of 
others,  and  no  single  measure  can  provide  a  clear  performance  target  or  focus  at¬ 
tention  on  all  the  critical  areas  of  the  business/program.  Managers  need  a  balanced 
presentation  of  both  financial  and  operational  measures. 

The  Balanced  Scorecard,  as  shown  in  Figure  B-l,  is  a  performance  management 
system  that  translates  mission  and  strategy  into  objectives  and  measures.  It  allows 
managers  to  look  at  their  business  from  four  important  perspectives,  and  it  pro¬ 
vides  answers  to  four  basic  questions: 

♦  How  do  customers  see  us  (customer  perspective)? 

♦  How  do  we  look  to  shareholders  (financial  perspective)? 

♦  What  must  we  excel  at  (internal  perspective)? 

♦  Can  we  continue  to  improve  and  create  value  (innovation,  learning,  and 
growth  perspective)? 


1  “The  Balanced  Scorecard — Measures  That  Drive  Performance,”  Robert  S.  Kaplan  and  David 
P.  Norton,  Harvard  Business  Review,  January-February  1992. 
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Figure  B-l.  The  Balanced  Scorecard 


The  scorecard  provides  a  framework — a  language — for  communicating  mission 
and  strategy;  it  uses  measurement  to  inform  employees  about  the  drivers  of  cur¬ 
rent  and  future  success.  By  clearly  defining  the  outcomes  the  organization  desires 
and  the  drivers  of  those  outcomes,  senior  managers  can  channel  the  energies, 
abilities,  and  specific  knowledge  of  people  throughout  the  organization  toward 
achieving  the  long-term  goals.2 

While  giving  senior  managers  information  from  the  four  perspectives,  the  bal¬ 
anced  scorecard  minimizes  information  overload  by  limiting  the  number  of  meas¬ 
ures  used.  Organizations  rarely  suffer  from  having  too  few  measures.  More 
commonly,  new  measures  are  added  whenever  an  external  priority  changes  or  an 
employee  or  consultant  make  a  worthwhile  suggestion.  More  than  one  program 
team  member  told  us,  “too  many  metrics,  too  few  measures  of  value.”  The  bal¬ 
anced  scorecard  forces  managers  to  focus  on  the  handful  of  measures  that  are  the 
most  critical. 

The  scorecard  brings  together,  in  a  single  management  report,  many  of  the  seem¬ 
ingly  disparate  elements  of  an  organization’s  competitive  agenda:  shortening  re¬ 
sponse  time,  managing  for  the  long  term,  improving  quality,  emphasizing  and 
rewarding  teamwork,  being  customer  oriented,  and  reducing  product  launch  times. 
The  scorecard  also  guards  against  sub-optimization.  By  forcing  senior  managers 
to  consider  all  the  operational  measures  together,  the  balanced  scorecard  lets  them 
see  whether  improvement  in  one  area  may  have  been  achieved  at  the  expense  of 
another. 


2  The  Balanced  Scorecard,  Translating  Strategy  into  Action,  Robert  S.  Kaplan  and  David  P. 
Norton,  Harvard  Business  School  Press,  1996. 
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A  Strategic  Management  Model 

The  balanced  scorecard  is  more  than  a  tactical  or  operational  measurement  exer¬ 
cise.  Innovative  companies  are  using  the  scorecard  as  a  strategic  management 
system,  to  manage  strategy  for  the  long  term  and  motivate  breakthrough  im¬ 
provements  in  such  critical  areas  as  product,  process,  customer,  and  market  devel¬ 
opment. 

The  scorecard  has  its  greatest  impact  when  it  is  used  to  drive  organizational 
change.3  Senior  executives  establish  targets  for  the  scorecard  measures  that,  if 
achieved,  will  transform  the  company.  Rather  than  just  applying  fundamental 
process  redesign  to  any  local  process  where  gains  might  be  easily  obtained,  mana¬ 
gerial  efforts  are  directed  to  improving  and  reengineering  processes  that  are  criti¬ 
cal  for  the  organization’s  strategic  success.  The  target-setting  management  process 
enables  the  organization  to 

♦  quantify  the  long-term  outcomes  it  wishes  to  achieve, 

♦  identify  mechanisms  and  provide  resources  for  achieving  those  outcomes,  and 

♦  establish  short-term  milestones  for  financial  and  nonfinancial  measures  on  the 
scorecard. 


Strategy 


The  scorecard  process  starts  with  a  senior  executive  management  team  working 
together  to  translate  strategy  into  specific  strategic  objectives.  The  team  develops 
an  initial  scorecard  with  fairly  narrow  objectives  to  gain  clarification,  consensus, 
and  focus  on  the  strategy,  and  to  communicate  that  strategy  throughout  the  organi¬ 
zation.  The  team  then  uses  the  measurement  focus  of  the  scorecard  to4 

♦  align  departmental  and  personal  goals  with  strategy, 

♦  link  strategic  objectives  to  long-term  targets  and  annual  budgets, 

♦  plan,  set  targets,  and  align  strategic  initiatives, 

♦  perform  periodic  and  systematic  strategic  reviews,  and 

♦  obtain  feedback  to  learn  about  and  improve  strategy. 

A  strategic  perspective  provides  criteria  for  deciding  the  issues  or,  at  least,  focuses 
the  debate  on  what  the  relevant  criteria  are.  Difficult  tactical  issues  are  often  more 
effectively  resolved  when  the  strategic  perspective  is  clear.  A  strategic  approach 


3  Ibid.,  p.  14. 

4  Ibid.,  p.  19. 
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enlarges  the  field  of  view,  so  that  tactical  questions  seldom  stand  alone.  They  are 
related  to  other  questions  that  eventually  lead  to  some  fundamental  issue  regard¬ 
ing  the  organization’s  ability  to  achieve  its  desired  outcomes.  The  most  effective 
organizations  over  the  long  term  devote  time  and  attention  to  resolving  tactical 
issues  and  questions  within  the  organization’s  broader  strategic  context. 

Customer  Perspective:  How  Do  Customers  See  Us? 

A  common  objective  today  is  delivering  value  to  the  customer.  How  an  organiza¬ 
tion  or  program  is  performing  from  the  customers’  perspective  has  become  a  pri¬ 
ority  for  top  leadership.  The  balanced  scorecard  requires  managers  to  translate 
their  general  mission  statement  on  customer  service  into  specific  measures  that 
reflect  the  factors  that  really  matter  to  customers. 

Customers’  concerns  tend  to  fall  into  four  categories:  time,  quality,  performance 
and  service,  and  cost.  To  put  the  scorecard  to  work,  goals  should  be  articulated  for 
time,  quality,  and  performance  and  service  and  then  translated  into  specific  meas¬ 
ures.  For  example,  lead-time  measures  the  time  required  to  meet  the  customers’ 
needs.  For  existing  products  and  services,  lead-time  can  be  measured  from  the 
time  an  order  is  received  to  the  time  the  product  or  service  is  delivered  to  the 
customer.  For  new  products,  lead-time  represents  the  time  to  market,  or  how  long 
it  takes  to  bring  a  new  product  from  the  product  definition  stage  to  the  start  of 
field  testing.  Quality  measures  the  defect  level  of  incoming  products  as  perceived 
and  measured  by  the  customer.  Quality  could  also  measure  on-time  delivery,  a 
function  of  the  accuracy  of  delivery  forecasts.  The  combination  of  performance 
and  service  measures  how  products  or  services  contribute  to  creating  value  for 
customers. 

In  addition  to  measures  of  time,  quality,  and  performance  and  service,  it  is  of 
course  important  to  remain  sensitive  to  cost.  Price  is  only  one  component  of  the 
cost  incurred  when  dealing  with  suppliers.  Other  supplier-driven  costs  range  from 
ordering,  scheduling  delivery,  and  paying  for  materials,  to  receiving,  inspecting, 
handling,  and  storing  the  materials.  They  also  reflect  the  scrap,  rework,  and  obso¬ 
lescence  caused  by  the  materials,  as  well  as  schedule  disruptions  (expediting  and 
value  of  lost  output)  from  incorrect  deliveries.  An  excellent  supplier  may  charge  a 
higher  unit  price  for  products/materials  than  other  vendors  but  nonetheless  be  a 
lower-cost  supplier  because  it  can  deliver  defect-free  products  in  exactly  the  right 
quantities  at  exactly  the  right  time  directly  to  the  production  process  and  can 
minimize  the  administrative  hassles  of  ordering,  invoicing,  and  paying  for  materi¬ 
als  through  electronic  data  interchange. 

Internal  Business  Perspective:  What  Must  We  Excel  At? 

Customer-based  measures  are  important,  but  they  must  be  translated  into  meas¬ 
ures  of  what  an  organization  must  do  internally  to  meet  its  customers’  expecta¬ 
tions.  After  all,  excellent  customer  performance  derives  from  processes,  decisions, 


B-4 


The  Balanced  Scorecard 


and  actions  occurring  throughout  an  organization.  Managers  need  to  focus  on 
those  critical  internal  operations  that  enable  them  to  satisfy  customer  needs  and 
requirements. 

Traditional  performance  measurement  systems — even  those  that  use  mostly  non- 
financial  indicators — focus  on  improving  the  cost,  quality,  and  cycle  times  of  ex¬ 
isting  processes.  The  balanced  scorecard  highlights  those  processes  that  are  most 
critical  for  achieving  breakthrough  performance  for  customers  and  stakeholders. 
Often  this  identification  reveals  entirely  new  internal  processes  that  the  organiza¬ 
tion  must  excel  at  for  its  strategy  to  be  successful. 

Internal  measures  should  attempt  to  capture  an  organization’s  core  competen¬ 
cies — the  critical  skills  and  technologies  needed  to  ensure  continued  success.  To 
achieve  goals,  managers  must  devise  measures  that  are  influenced  by  employees’ 
actions.  Since  much  of  the  action  takes  place  at  the  department/integrated  Product 
Team  level,  managers  need  to  cascade  objectives  to  lower  levels.  That  way,  the 
measures  link  top  management’s  judgment  about  key  internal  processes  and  com¬ 
petencies  to  the  actions  taken  by  individuals  that  affect  strategic  objectives.  This 
linkage  ensures  that  employees  at  lower  levels  in  the  organization  have  clear  tar¬ 
gets  for  actions,  decisions,  and  improvement  activities  that  will  contribute  to  the 
overall  mission. 

Learning  and  Growth  (Innovation)  Perspective:  Can  We  Continue 
to  Improve  and  Create  Value? 

A  major  issue  in  determining  innovation  and  learning  is  the  organization  mem¬ 
bers’  understanding  of  “what  we  are  here  to  do,”  and  whether  learning,  improve¬ 
ment,  and  adoption  of  more  effective  practices  is  part  of  the  understanding  of  the 
requirements  of  their  role.  An  organizations’  ability  to  innovate,  improve,  and 
learn  ties  directly  to  its  value. 

One  company  has  validated  the  cause-and-effect  relationships  in  its  balanced 
scorecard  by  measuring  the  strength  of  the  linkages  among  measures  in  the  per¬ 
spectives.5  The  company  found  significant  correlations  between  employees’  mo¬ 
rale  (a  measure  in  the  leaming-and-growth  perspective)  and  customer  satisfaction 
(an  important  customer  perspective  measure).  Customer  satisfaction,  in  turn,  was 
correlated  with  faster  payment  of  invoices,  a  relationship  that  led  to  a  substantial 
reduction  in  accounts  receivable  and  hence  a  higher  return  on  capital  employed. 
Organizations  have  also  found  correlations  between  employees’  morale  and  the 
number  of  suggestions  made  by  employees  (both  leaming-and-growth  measures), 
as  well  as  between  an  increased  number  of  suggestions  and  lower  rework  (an  in¬ 
ternal  business  process  measure).  Evidence  of  such  strong  correlations  helps  to 
confirm  the  organization’s  business  strategy.  If  expected  correlations  are  not 

5  “The  Balanced  Scorecard — Measures  That  Drive  Performance,”  Robert  S.  Kaplan  and  David 
P.  Norton,  Harvard  Business  Review ,  January-February  1992. 
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found  over  time,  that  should  indicate  to  senior  management  that  the  theory  un¬ 
derlying  the  organization’s  strategy  may  not  be  working  as  anticipated. 

Financial  Perspective:  How  Do  We  Look  to  Shareholders? 

Is  the  financial  perspective  even  relevant  for  driving  the  long-term  performance  of 
a  DoD  organization?  Several  critics  of  industry  performance  measurement  sys¬ 
tems  have  advocated  scrapping  financial  measures  entirely  for  measuring  business 
unit  performance,  arguing  that  in  today’s  technologically  and  customer-driven 
global  competition,  financial  measures  are  poor  guidelines  for  success.  Rather 
than  managing  the  bottom  line  to  facilitate  short-term  savings,  critics  argue  for 
investing  in  the  long  term  (customers,  employees,  systems).  The  most  commonly 
used  results  measures  in  product  development  are  schedule  and  cost.  But  the  fact 
that  a  program  is  six  months  late  and  $2  million  over  budget  doesn’t  tell  anyone 
what  went  wrong  or  what  to  do  differently.6 

Financial  objectives  and  measures  must  define  the  financial  performance  expected 
from  the  strategy.  The  hard  truth  is  that  if  improved  performance  fails  to  be  re¬ 
flected  in  the  bottom  line,  managers  need  to  reexamine  the  basic  assumptions  of 
their  strategy  and  mission.  Performance  measures  are  the  glue  that  binds  business 
strategy  to  operational  behavior.  Therefore,  it  is  critical  to  effectively  link  the  fi¬ 
nancial  and  strategic  objectives  to  the  operational  cost  drivers  that  can  be  man¬ 
aged  at  the  functional  or  process  level.7 

Summary 

An  organization’s  measurement  system  strongly  affects  the  behavior  of  people 
both  inside  and  outside  the  organization. 

Performance  measures  should  be  used  to  articulate  the  strategy  of  the  business,  to 
communicate  the  strategy  of  the  business,  and  to  help  align  individual,  organiza¬ 
tional,  and  cross-departmental  initiatives  to  achieve  a  common  goal.  The  four  per¬ 
spectives  of  the  scorecard  permit  a  balance  between  short  and  long  term 
objectives,  between  outcomes  desired  and  the  performance  drivers  of  those  out¬ 
comes,  and  between  hard  objective  measures  and  softer,  more  subjective  meas¬ 
ures.  While  the  multiplicity  of  measures  on  a  scorecard  may  seem  confusing, 
properly  constructed  scorecards  have  a  unity  of  purpose,  since  all  the  measures  are 
directed  toward  achieving  an  integrated  strategy.8 


6  “How  the  Right  Measures  Help  Teams  Excel,”  Christopher  Meyer,  Harvard  Business  Re- 
view ,  May- June  1994. 

7  Gary  Hudson,  Manager  of  Advanced  Cost  Management,  AlliedSignal,  Inc.,  1997. 

8  The  Balanced  Scorecard ,  Translating  Strategy  into  Action ,  Robert  S.  Kaplan  and  David  P. 
Norton,  Harvard  Business  School  Press,  1996. 
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Appendix  C 

Abbreviations 


ACA 

American  Compensation  Association 

ACATI 

Acquisition  Category  I 

ACV 

average  cost  variance 

ANOVA 

analysis  of  variance 

APB 

Acquisition  Program  Baseline 

APQC 

American  Productivity  and  Quality  Center 

ASD(C3I) 

Assistant  Secretary  of  Defense  for  Command,  Control, 
Communications,  and  Intelligence 

ASV 

average  schedule  variance 

CAE 

Component  Acquisition  Executive 

CATV 

cost  as  an  independent  variable 

CEO 

Chief  Executive  Officer 

COTS 

commercial-off-the-shelf 

CV 

cost  variance 

DoD 

Department  of  Defense 

DSCPD 

Defense  System  Cost  Performance  Database 

DUSD(AR) 

Deputy  Under  Secretary  of  Defense  for  Acquisition  Reform 

EMD 

engineering,  manufacturing,  and  development 

FASA 

Federal  Acquisition  Streamlining  Act 

GPRA 

Government  Performance  and  Results  Act 

me 

International  Benchmarking  Clearinghouse 

IMPROSHARE 

Improved  Productivity  through  Sharing 

IPT 

Integrated  Product  Team 

IT 

information  technology 

LMI 

Logistics  Management  Institute 

MAIS 

major  automated  information  system 

MILCON 

construction 
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O&M 

operations  and  maintenance 

OPM 

Office  of  Personnel  Management 

P&C 

Property  and  Casualty 

PBOs 

performance-based  organizations 

PEO 

Program  Executive  Officer 

PM 

Program  Manager 

PMT 

program  management  team 

QRMC 

Quadrennial  Review  of  Military  Compensation 

RDT&E 

research,  development,  test,  and  evaluation 

SARs 

Selected  Acquisition  Reports 

SETA 

science,  engineering,  technical,  and  administrative 

SV 

schedule  variance 

U.S.C. 

United  States  Code 

USD(A&T) 

Under  Secretary  of  Defense  for  Acquisition  and 
Technology 

USM&R 

U.S.  Marketing  &  Refining  Division 

VA 

Veterans  Affairs 
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